``` ├── .gitignore ├── Dockerfile ├── README.md ├── class_Legendary_model.pkl ├── dashboards/ ├── metrics/ ├── drift_detail_metrics.json ├── drift_metrics.json ├── drift_summary_metrics.json ├── roi_dashboard.json ├── datasets/ ├── bank_num.csv ├── bank_num_old.csv ``` ## /.gitignore ```gitignore path="/.gitignore" .vscode # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ pip-wheel-metadata/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py,cover .hypothesis/ .pytest_cache/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # PEP 582; used by e.g. github.com/David-OConnor/pyflow __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # MacOS special files .DS_Store .DS_Store? # Catboost folder catboost_info/ src/catboost_info/ ``` ## /Dockerfile ``` path="/Dockerfile" FROM python:3.9.6-slim-buster RUN apt-get update && apt-get install -y git gcc cron # Add crontab file in the cron directory ADD ./src/crontab/crontab /etc/cron.d/crontab # Give execution rights on the cron job RUN chmod 0644 /etc/cron.d/crontab # Create the log file to be able to run tail RUN touch /var/log/cron.log WORKDIR /app COPY requirements.txt requirements.txt COPY src/compute_metrics_script.py compute_metrics_script.py COPY src/gen_data_and_simulate_drift.py gen_data_and_simulate_drift.py COPY src/fake_data_script.py fake_data_script.py COPY src/training_script.py training_script.py COPY src/roi.py roi.py COPY src/class_Legendary_model.pkl class_Legendary_model.pkl COPY datasets/ datasets/ COPY models/ models/ RUN pip install --no-cache-dir -r requirements.txt RUN /usr/bin/crontab /etc/cron.d/crontab # Run the command on container startup CMD ["cron", "-f"] # RUN adduser app && chown -R app /app # USER app # # RUN python -m venv /app/.venv # ENV PATH="/app/.venv/bin:$PATH" # # COPY requirements.txt /app/requirements.txt # COPY main.py /app/main.py # COPY kidney_disease.pkl /app/kidney_disease.pkl # # RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt # # EXPOSE 8000 # CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] # CMD ["while true; do sleep 15; done"] ``` ## /README.md # Pulsar.ML Local demo This repository contains an example use-case demonstrating how to utilize [pulsar-data-collection](https://github.com/Rocket-Science-Development/pulsar_data_collection) and [pulsar-metrics](https://github.com/Rocket-Science-Development/pulsar_metrics) to implement model monitoring and performance management. ## Quickstart 1. Make sure that you have the Docker and Docker compose installed and execute the following commands: ```bash export GRAFANA_USERNAME=admin; export GRAFANA_PASSWORD=pass123; export DB_USER=admin; export DB_PASSWORD=pass123; docker-compose up --build ``` 2. Open a web browser and go to localhost:3000. Enter the Grafana credentials set in the previous step. ## Pulsar.ML Architecture ![Architecture](images/PulsarDemoFlowDiagram_v1.jpeg) ## Demo Components ![Demo workflow](images/pulsar-demo-workflow.png) Here is a description of the steps within the workflow 1. Data is captured from an inference container or a notebook using [pulsar-data-collection](https://github.com/Rocket-Science-Development/pulsar_data_collection) 2. Collected data point, predictions, and other relevant configured metadata are written into an index inside influxdb 3. `compute-metrics` service will query the latest entries from the database then leverage [pulsar-metrics](https://github.com/Rocket-Science-Development/pulsar_metrics) in order to compute the different metrics. 4. All computed metrics are then written to another index in InfluxDB for display in Grafana. ## About [Pulsar.ML](https://pulsar.ml/) Pulsar.ML is an application designed to assist with monitoring your machine learning models and gaining powerful insights into their performance. We have released two open-source packages: - [pulsar-data-collection](https://github.com/Rocket-Science-Development/pulsar_data_collection) : A lightweight Python SDK for collecting features, predictions, and metadata from an ML model serving code or microservice. - [pulsar-metrics](https://github.com/Rocket-Science-Development/pulsar_metrics) : A library for evaluating and monitoring data and concept drift with an extensive set of metrics. It also provides the flexibility to use custom metrics defined by the user. We also created [pulsar demo](https://github.com/Rocket-Science-Development/pulsar_demo) to showcase an example use-case that demonstrates how to leverage both packages for model monitoring and performance management. For further interaction with our community, Please join our [slack channel](https://pulsarml.slack.com) Powered by [Rocket Science Development](https://rocketscience.one/) ## Contributing 1. Fork this repository, develop, and test your changes 2. open an issue 3. Submit a pull request with a reference to the issue ## /class_Legendary_model.pkl Binary file available at https://raw.githubusercontent.com/Rocket-Science-Development/pulsar_demo/refs/heads/main/class_Legendary_model.pkl ## /dashboards/metrics/drift_detail_metrics.json ```json path="/dashboards/metrics/drift_detail_metrics.json" { "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": 2, "links": [], "liveNow": false, "panels": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "gridPos": { "h": 7, "w": 4, "x": 0, "y": 0 }, "id": 4, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "", "mode": "html" }, "pluginVersion": "9.5.3", "transparent": true, "type": "text" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#9d0bde", "mode": "fixed" }, "decimals": 3, "displayName": "Max:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 }, "id": 6, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "max" ], "fields": "", "values": false }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "9.5.3", "targets": [ { "alias": "", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' ", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "purple", "mode": "fixed" }, "decimals": 3, "displayName": "Min:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 4, "w": 4, "x": 8, "y": 0 }, "id": 7, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "min" ], "fields": "", "values": false }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "9.5.3", "targets": [ { "alias": "", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' ", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "super-light-purple", "mode": "fixed" }, "displayName": "#Count:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 4, "w": 4, "x": 12, "y": 0 }, "id": 10, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "min" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "9.5.3", "targets": [ { "alias": "", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT COUNT(metric_value) FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "light-purple", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 3, "w": 4, "x": 8, "y": 4 }, "id": 5, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "/^Time$/", "values": false }, "text": { "titleSize": 20, "valueSize": 20 }, "textMode": "auto" }, "pluginVersion": "9.5.3", "targets": [ { "alias": "", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT time, metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' ORDER BY time DESC LIMIT 1", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Last Value Inserted at", "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#b0aaf6", "mode": "fixed" }, "displayName": "Change", "mappings": [], "thresholds": { "mode": "percentage", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 70 }, { "color": "red", "value": 85 } ] }, "unit": "percent" }, "overrides": [] }, "gridPos": { "h": 4, "w": 4, "x": 12, "y": 4 }, "id": 8, "options": { "orientation": "auto", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "showThresholdLabels": false, "showThresholdMarkers": true, "text": { "titleSize": 17, "valueSize": 40 } }, "pluginVersion": "9.5.3", "targets": [ { "alias": "", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' ", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [ { "id": "reduce", "options": { "includeTimeField": false, "labelsToFields": false, "mode": "seriesToRows", "reducers": [ "max", "range" ] } }, { "id": "calculateField", "options": { "alias": "RangeDiv", "binary": { "left": "Range", "operator": "/", "reducer": "sum", "right": "Max" }, "mode": "binary", "reduce": { "reducer": "sum" } } }, { "id": "calculateField", "options": { "alias": "Change%", "binary": { "left": "RangeDiv", "operator": "*", "reducer": "sum", "right": "100" }, "mode": "binary", "reduce": { "reducer": "sum" }, "replaceFields": true } } ], "transparent": true, "type": "gauge" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "custom": { "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "scaleDistribution": { "type": "linear" } } }, "overrides": [] }, "gridPos": { "h": 16, "w": 9, "x": 0, "y": 7 }, "id": 9, "options": { "calculate": true, "calculation": { "xBuckets": { "mode": "size" }, "yBuckets": { "mode": "size" } }, "cellGap": 1, "color": { "exponent": 0.5, "fill": "dark-orange", "mode": "scheme", "reverse": false, "scale": "exponential", "scheme": "Greens", "steps": 64 }, "exemplars": { "color": "rgba(255,0,255,0.7)" }, "filterValues": { "le": 1e-9 }, "legend": { "show": true }, "rowsFrame": { "layout": "auto" }, "tooltip": { "show": true, "yHistogram": false }, "yAxis": { "axisPlacement": "left", "reverse": false } }, "pluginVersion": "9.5.3", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT time, metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Heatmap for Metrics vs Time", "transparent": true, "type": "heatmap" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#8d82ff", "mode": "fixed" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "right", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 25, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "smooth", "lineStyle": { "dash": [ 10, 10 ], "fill": "dash" }, "lineWidth": 4, "pointSize": 6, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": true, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "decimals": 2, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [ { "matcher": { "id": "byName", "options": "threshold" }, "properties": [ { "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }, { "id": "custom.axisPlacement", "value": "left" }, { "id": "unit", "value": "percentunit" } ] } ] }, "gridPos": { "h": 15, "w": 15, "x": 9, "y": 8 }, "id": 2, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "$metric_name value", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" }, { "alias": "threshold", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "hide": false, "query": "SELECT threshold FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "B", "resultFormat": "time_series" } ], "title": "Drift Detail for Feature - $feature_name and Metric - $metric_name", "transparent": true, "type": "timeseries" } ], "refresh": "", "revision": 1, "schemaVersion": 38, "style": "dark", "tags": [], "templating": { "list": [ { "current": { "selected": false, "text": "drift", "value": "drift" }, "hide": 0, "includeAll": false, "multi": false, "name": "type", "options": [ { "selected": true, "text": "drift", "value": "drift" } ], "query": "drift", "queryValue": "", "skipUrlSync": false, "type": "custom" }, { "current": { "selected": false, "text": "1", "value": "1" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "model_id", "options": [], "query": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "2", "value": "2" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "model_version", "options": [], "query": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": true, "text": "HP", "value": "HP" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"feature_name\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "feature_name", "options": [], "query": "SELECT distinct(\"feature_name\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "manwu", "value": "manwu" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"metric_name\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "metric_name", "options": [], "query": "SELECT distinct(\"metric_name\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" } ] }, "time": { "from": "now-7d", "to": "now" }, "timepicker": {}, "timezone": "", "title": "DriftDetailDashboard", "uid": "Zntdh-tVz", "version": 8, "weekStart": "" } ``` ## /dashboards/metrics/drift_metrics.json ```json path="/dashboards/metrics/drift_metrics.json" { "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": 4, "links": [ { "asDropdown": false, "icon": "external link", "includeVars": false, "keepTime": true, "tags": [], "targetBlank": true, "title": "ROI", "tooltip": "", "type": "link", "url": "http://localhost:3000/goto/CdGv5VkSg?orgId=1" }, { "asDropdown": false, "icon": "doc", "includeVars": false, "keepTime": false, "tags": [], "targetBlank": true, "title": "GitHub", "tooltip": "", "type": "link", "url": "https://github.com/Rocket-Science-Development/pulsar_demo" } ], "liveNow": false, "panels": [ { "collapsed": false, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 18, "panels": [], "title": "Drift", "type": "row" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "gridPos": { "h": 6, "w": 3, "x": 0, "y": 1 }, "id": 26, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "", "mode": "html" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics where type = '$type' and feature = '$feature_name' and metric_name = 'CvM' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transparent": true, "type": "text" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-purple", "mode": "fixed" }, "decimals": 4, "displayName": "Max:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 4, "y": 1 }, "id": 31, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "max" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT MAX(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-purple", "mode": "fixed" }, "decimals": 4, "displayName": "Min:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 7, "y": 1 }, "id": 32, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "min" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT MIN(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "dark-purple", "mode": "fixed" }, "decimals": 4, "displayName": "Mean:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 10, "y": 1 }, "id": 33, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "mean" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT MEAN(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Time Series graphs for all metrics for the selected Feature. ", "fieldConfig": { "defaults": { "color": { "fixedColor": "purple", "mode": "fixed" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 10, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": true, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [ { "matcher": { "id": "byRegexp", "options": "/TRUE$/" }, "properties": [ { "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } } ] }, { "matcher": { "id": "byRegexp", "options": "/FALSE$/" }, "properties": [ { "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } } ] } ] }, "gridPos": { "h": 11, "w": 9, "x": 15, "y": 1 }, "id": 28, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "$feature_name-Prediction=TRUE", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT \"$feature_name\" FROM prediction where y_pred =1", "rawQuery": true, "refId": "A", "resultFormat": "time_series" }, { "alias": "$feature_name-Prediction=FALSE", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "hide": false, "query": "SELECT \"$feature_name\" FROM prediction where y_pred =0", "rawQuery": true, "refId": "B", "resultFormat": "time_series" }, { "alias": "$feature_name-Moving Avg", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "hide": false, "query": "SELECT moving_average(mean(\"$feature_name\"), 2) FROM prediction WHERE $timeFilter GROUP BY time($__interval) fill(null)", "rawQuery": true, "refId": "D", "resultFormat": "time_series" } ], "title": "Comparison", "transformations": [], "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#9678d4", "mode": "fixed" }, "decimals": 4, "displayName": "Std Dev:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 4, "y": 4 }, "id": 35, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "stdDev" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT STDDEV(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter ", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#9678d4", "mode": "fixed" }, "decimals": 4, "displayName": "50th percentile:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 7, "y": 4 }, "id": 34, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "variance" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT median(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "#9678d4", "mode": "fixed" }, "displayName": "# Metrics Drifted:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 10, "y": 4 }, "id": 36, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT COUNT(DISTINCT(metric_name)) FROM metrics WHERE drift_status='True' AND metric_type = '$type' and feature_name = '$feature_name' and model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "align": "auto", "cellOptions": { "type": "auto" }, "inspect": false }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 5, "w": 3, "x": 4, "y": 7 }, "id": 37, "options": { "cellHeight": "sm", "footer": { "countRows": false, "fields": "", "reducer": [ "sum" ], "show": false }, "showHeader": true }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT DISTINCT(metric_name) FROM metrics WHERE drift_status='True' AND metric_type = '$type' and feature_name = '$feature_name' and model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [ { "id": "organize", "options": { "excludeByName": { "Time": true }, "indexByName": {}, "renameByName": { "metrics.distinct": "Drifted Metrics" } } } ], "transparent": true, "type": "table" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "", "fieldConfig": { "defaults": { "color": { "fixedColor": "super-light-purple", "mode": "fixed" }, "decimals": 0, "displayName": "# Predictions:", "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "short" }, "overrides": [] }, "gridPos": { "h": 3, "w": 3, "x": 7, "y": 7 }, "id": 38, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "variance" ], "fields": "", "values": true }, "text": { "titleSize": 20, "valueSize": 45 }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT COUNT(\"$feature_name\") FROM prediction where model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transformations": [], "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 2, "w": 3, "x": 10, "y": 7 }, "id": 29, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "firstNotNull" ], "fields": "/^Time$/", "values": false }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT time, metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and model_id = '$model_id' and model_version = '$model_version' ORDER BY time ASC LIMIT 1", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "First Value Inserted on", "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 2, "w": 3, "x": 10, "y": 9 }, "id": 30, "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "/^Time$/", "values": false }, "textMode": "auto" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT time, metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and model_id = '$model_id' and model_version = '$model_version' ORDER BY time DESC LIMIT 1", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Last Value Inserted on", "transparent": true, "type": "stat" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Brown Forsythe test metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=bftest" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 0, "y": 12 }, "id": 24, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "Bf test", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'bftest' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Brown Forsythe Test", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Ttest metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=ttest" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 6, "y": 12 }, "id": 19, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "pluginVersion": "9.1.6", "targets": [ { "alias": "T-test", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'ttest' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Ttest", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Manwu metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=manwu" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 12, "y": 12 }, "id": 22, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "pluginVersion": "9.2.5", "targets": [ { "alias": "Mann WU test", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'manwu' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Mann-Whitney U Test", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Kolmogorov Smirnov metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=ks_2samp" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 18, "y": 12 }, "id": 20, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "KS", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'ks_2samp' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Kolmogorov Smirnov", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Wasserstein metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=wasserstein" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 0, "y": 20 }, "id": 14, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "Wasserstein", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'wasserstein' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Wasserstein Distance", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Levene metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=levene" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 6, "y": 20 }, "id": 23, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "pluginVersion": "9.1.6", "targets": [ { "alias": "Levene", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'levene' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Levene Test", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Kullback-Leibler metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=kl" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 12, "y": 20 }, "id": 21, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "KL", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'kl' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Kullback-Leibler Divergence", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Cramer-von Mises metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=CvM" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 18, "y": 20 }, "id": 25, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "CvM", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'CvM' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Cramer-von Mises", "transparent": true, "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Population Stability Index metric corresponding to the Feature - $feature_name. Click on the data points to navigate to more detailed view.", "fieldConfig": { "defaults": { "color": { "mode": "continuous-BlPu" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 20, "gradientMode": "scheme", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 3, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "links": [ { "targetBlank": true, "title": "DriftDetailDashboard", "url": "http://localhost:3000/d/Zntdh-tVz/driftdetaildashboard?orgId=1&var-type=$type&var-model_id=$model_id&var-model_version=$model_version&var-feature_name=$feature_name&var-metric_name=CvM" } ], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 8, "w": 6, "x": 0, "y": 28 }, "id": 39, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "psi", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_type = '$type' and feature_name = '$feature_name' and metric_name = 'psi' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Population Stability Index", "transparent": true, "type": "timeseries" }, { "collapsed": true, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 36 }, "id": 16, "panels": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 0, "y": 32 }, "id": 6, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics where type = '$type' and feature = '$feature_name' and metric_name = 'precision'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Precision", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 12, "y": 32 }, "id": 9, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'precision'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Precision", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 0, "y": 47 }, "id": 5, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'recall'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Recall", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 12, "y": 47 }, "id": 8, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'f1'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "F1", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 0, "y": 62 }, "id": 7, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'brier'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Brier", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 12, "y": 62 }, "id": 11, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'brier'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "Brier", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 0, "y": 77 }, "id": 10, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'mae'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "MAE", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 12, "y": 77 }, "id": 12, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'mape'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "MAPE", "type": "timeseries" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 0, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineInterpolation": "linear", "lineWidth": 1, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green" }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 15, "w": 12, "x": 0, "y": 92 }, "id": 13, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics_db where type = '$type' and feature = '$feature_name' and metric_name = 'r2'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "R2", "type": "timeseries" } ], "title": "Perfomence", "type": "row" } ], "refresh": false, "revision": 1, "schemaVersion": 38, "style": "dark", "tags": [], "templating": { "list": [ { "current": { "selected": false, "text": "drift", "value": "drift" }, "description": "Type of the metric", "hide": 0, "includeAll": false, "label": "Type", "multi": false, "name": "type", "options": [ { "selected": true, "text": "drift", "value": "drift" } ], "query": "drift", "queryValue": "", "skipUrlSync": false, "type": "custom" }, { "allValue": "", "current": { "selected": true, "text": "housing", "value": "housing" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"feature_name\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "label": "Feature Name", "multi": false, "name": "feature_name", "options": [], "query": "SELECT distinct(\"feature_name\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "1", "value": "1" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "label": "Model Id", "multi": false, "name": "model_id", "options": [], "query": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "2", "value": "2" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$type'", "hide": 0, "includeAll": false, "label": "Model Version", "multi": false, "name": "model_version", "options": [], "query": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "CvM", "value": "CvM" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"metric_name\") FROM metrics where metric_type = '$type'", "hide": 2, "includeAll": false, "label": "Metric Name", "multi": false, "name": "metric_name", "options": [], "query": "SELECT distinct(\"metric_name\") FROM metrics where metric_type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" } ] }, "time": { "from": "2023-08-29T03:20:02.475Z", "to": "2023-08-30T16:50:01.810Z" }, "timepicker": {}, "timezone": "", "title": "Drift Dashboard", "uid": "y4-ZCkG4k", "version": 7, "weekStart": "" } ``` ## /dashboards/metrics/drift_summary_metrics.json ```json path="/dashboards/metrics/drift_summary_metrics.json" { "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": 4, "links": [], "liveNow": false, "panels": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "gridPos": { "h": 8, "w": 4, "x": 0, "y": 0 }, "id": 4, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "", "mode": "html" }, "pluginVersion": "9.3.2", "transparent": true, "type": "text" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "custom": { "align": "auto", "displayMode": "auto", "inspect": false }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [] }, "gridPos": { "h": 23, "w": 20, "x": 4, "y": 0 }, "id": 2, "options": { "footer": { "enablePagination": true, "fields": "", "reducer": [ "sum" ], "show": false }, "frameIndex": 1, "showHeader": true }, "pluginVersion": "9.3.2", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "hide": false, "query": "SELECT feature, value FROM metrics where type = 'drift' and metric_name = '$metric_name' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "table" } ], "title": "Feature Summary", "transformations": [ { "id": "extractFields", "options": { "format": "auto", "replace": false, "source": "value" } }, { "id": "groupBy", "options": { "fields": { "feature": { "aggregations": [], "operation": "groupby" }, "value": { "aggregations": [ "max", "min", "range", "distinctCount", "changeCount" ], "operation": "aggregate" } } } }, { "id": "calculateField", "options": { "alias": "ChangeAmt", "binary": { "left": "value (range)", "operator": "/", "reducer": "sum", "right": "value (max)" }, "mode": "binary", "reduce": { "reducer": "sum" } } }, { "id": "calculateField", "options": { "alias": "FeatureChange%", "binary": { "left": "ChangeAmt", "operator": "*", "reducer": "sum", "right": "100" }, "mode": "binary", "reduce": { "reducer": "sum" } } }, { "id": "organize", "options": { "excludeByName": { "ChangeAmt": true, "value (range)": true }, "indexByName": { "ChangeAmt": 7, "FeatureChange%": 4, "feature": 0, "value (changeCount)": 6, "value (distinctCount)": 5, "value (max)": 1, "value (min)": 2, "value (range)": 3 }, "renameByName": { "feature": "Feature", "value (changeCount)": "ChangeCount", "value (distinctCount)": "DistinctCount", "value (max)": "Max", "value (min)": "Min" } } } ], "transparent": true, "type": "table" } ], "refresh": false, "schemaVersion": 37, "style": "dark", "tags": [], "templating": { "list": [ { "current": { "selected": false, "text": "CvM", "value": "CvM" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"metric_name\") FROM metrics where type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "metric_name", "options": [], "query": "SELECT distinct(\"metric_name\") FROM metrics where type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "1", "value": "1" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_id\") FROM metrics where type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "model_id", "options": [], "query": "SELECT distinct(\"model_id\") FROM metrics where type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "2", "value": "2" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_version\") FROM metrics where type = '$type'", "hide": 0, "includeAll": false, "multi": false, "name": "model_version", "options": [], "query": "SELECT distinct(\"model_version\") FROM metrics where type = '$type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "hide": 2, "name": "type", "query": "drift", "skipUrlSync": false, "type": "constant" } ] }, "time": { "from": "now-2d", "to": "now" }, "timepicker": {}, "timezone": "", "title": "DriftSummaryDashboard", "uid": "663ynapVk", "version": 2, "weekStart": "" } ``` ## /dashboards/metrics/roi_dashboard.json ```json path="/dashboards/metrics/roi_dashboard.json" { "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" } ] }, "description": "Dashboard for ROI display", "editable": true, "fiscalYearStartMonth": 0, "graphTooltip": 0, "id": 5, "links": [], "liveNow": false, "panels": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "gridPos": { "h": 7, "w": 4, "x": 0, "y": 0 }, "id": 1, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "", "mode": "html" }, "pluginVersion": "10.1.0", "targets": [ { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT value FROM metrics where type = '$type' and feature = '$feature_name' and metric_name = 'CvM' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "transparent": true, "type": "text" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "ROI metric", "fieldConfig": { "defaults": { "color": { "fixedColor": "purple", "mode": "fixed" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "fillOpacity": 80, "gradientMode": "none", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "lineWidth": 1, "scaleDistribution": { "type": "linear" }, "thresholdsStyle": { "mode": "off" } }, "links": [], "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "currencyUSD" }, "overrides": [] }, "gridPos": { "h": 12, "w": 20, "x": 4, "y": 0 }, "id": 2, "options": { "barRadius": 0.2, "barWidth": 0.96, "fullHighlight": false, "groupWidth": 0.7, "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "orientation": "auto", "showValue": "auto", "stacking": "none", "tooltip": { "mode": "single", "sort": "none" }, "xTickLabelRotation": 0, "xTickLabelSpacing": 0 }, "pluginVersion": "9.1.6", "targets": [ { "alias": "Return of Investment (in $)", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "query": "SELECT metric_value FROM metrics where metric_name = 'roi' and model_id = '$model_id' and model_version = '$model_version' and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series" } ], "title": "ROI", "transparent": true, "type": "barchart" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Confusion matrix showing the breakdown of TP, TN, FP, FN values", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "hideFrom": { "legend": false, "tooltip": false, "viz": false } }, "mappings": [] }, "overrides": [] }, "gridPos": { "h": 10, "w": 12, "x": 0, "y": 12 }, "id": 3, "options": { "displayLabels": [ "percent" ], "legend": { "displayMode": "list", "placement": "right", "showLegend": true, "values": [] }, "pieType": "pie", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "True Positive", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "orderByTime": "ASC", "policy": "autogen", "query": "SELECT count(\"y_pred\") FROM prediction where \"y_pred\"=1 and \"default\"=1 and $timeFilter", "rawQuery": true, "refId": "A", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] }, { "alias": "True Negative", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "hide": false, "orderByTime": "ASC", "policy": "autogen", "query": "SELECT count(\"y_pred\") FROM prediction where \"y_pred\"=0 and \"default\"=0 and $timeFilter", "rawQuery": true, "refId": "B", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] }, { "alias": "False Positive", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "hide": false, "orderByTime": "ASC", "policy": "autogen", "query": "SELECT count(\"y_pred\") FROM prediction where \"y_pred\"=1 and \"default\"=0 and $timeFilter", "rawQuery": true, "refId": "C", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] }, { "alias": "False Negative", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "hide": false, "orderByTime": "ASC", "policy": "autogen", "query": "SELECT count(\"y_pred\") FROM prediction where \"y_pred\"=0 and \"default\"=1 and $timeFilter", "rawQuery": true, "refId": "D", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] } ], "title": "Confusion Matrix", "transparent": true, "type": "piechart" }, { "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "description": "Performance matrices for getting a better understanding of how ROI is co-related with different parameters.", "fieldConfig": { "defaults": { "color": { "mode": "palette-classic" }, "custom": { "axisCenteredZero": false, "axisColorMode": "text", "axisLabel": "", "axisPlacement": "auto", "barAlignment": 0, "drawStyle": "line", "fillOpacity": 15, "gradientMode": "hue", "hideFrom": { "legend": false, "tooltip": false, "viz": false }, "insertNulls": false, "lineInterpolation": "smooth", "lineWidth": 2, "pointSize": 5, "scaleDistribution": { "type": "linear" }, "showPoints": "auto", "spanNulls": false, "stacking": { "group": "A", "mode": "none" }, "thresholdsStyle": { "mode": "off" } }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] }, "unit": "percentunit" }, "overrides": [] }, "gridPos": { "h": 10, "w": 12, "x": 12, "y": 12 }, "id": 4, "options": { "legend": { "calcs": [], "displayMode": "list", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "none" } }, "targets": [ { "alias": "Accuracy", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "orderByTime": "ASC", "policy": "autogen", "query": "SELECT metric_value FROM metrics where metric_name = 'accuracy' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "A", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] }, { "alias": "Precision", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "hide": false, "orderByTime": "ASC", "policy": "autogen", "query": "SELECT metric_value FROM metrics where metric_name = 'precision' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "B", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] }, { "alias": "Recall", "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "groupBy": [ { "params": [ "$__interval" ], "type": "time" }, { "params": [ "null" ], "type": "fill" } ], "hide": false, "orderByTime": "ASC", "policy": "autogen", "query": "SELECT metric_value FROM metrics where metric_name = 'recall' and model_id = '$model_id' and model_version = '$model_version'", "rawQuery": true, "refId": "C", "resultFormat": "time_series", "select": [ [ { "params": [ "value" ], "type": "field" }, { "params": [], "type": "mean" } ] ], "tags": [] } ], "title": "Performance Metrics", "transparent": true, "type": "timeseries" } ], "refresh": false, "schemaVersion": 38, "style": "dark", "tags": [], "templating": { "list": [ { "current": { "selected": false, "text": "custom", "value": "custom" }, "hide": 0, "includeAll": false, "label": "metric type", "multi": false, "name": "metric_type", "options": [ { "selected": true, "text": "custom", "value": "custom" } ], "query": "custom", "skipUrlSync": false, "type": "custom" }, { "current": { "selected": false, "text": "1", "value": "1" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$metric_type'", "hide": 0, "includeAll": false, "label": "model id", "multi": false, "name": "model_id", "options": [], "query": "SELECT distinct(\"model_id\") FROM metrics where metric_type = '$metric_type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" }, { "current": { "selected": false, "text": "2", "value": "2" }, "datasource": { "type": "influxdb", "uid": "PB6B4F2F1C736D27A" }, "definition": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$metric_type'", "hide": 0, "includeAll": false, "label": "model version", "multi": false, "name": "model_version", "options": [], "query": "SELECT distinct(\"model_version\") FROM metrics where metric_type = '$metric_type'", "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "type": "query" } ] }, "time": { "from": "2023-08-29T16:21:59.212Z", "to": "2023-08-30T16:21:59.212Z" }, "timepicker": {}, "timezone": "", "title": "ROI-dashboard", "uid": "b93ee885-457d-4298-b2d6-563bcb2efb42", "version": 6, "weekStart": "" } ``` ## /datasets/bank_num.csv ```csv path="/datasets/bank_num.csv" age,job,marital,education,default,balance,housing,loan,contact,day,month,duration,campaign,pdays,previous,poutcome,deposit 59,0,1,1,no,2343,1,0,2,5,8,1042,1,-1,0,3,1 56,0,1,1,no,45,0,0,2,5,8,1467,1,-1,0,3,1 41,9,1,1,no,1270,1,0,2,5,8,1389,1,-1,0,3,1 55,7,1,1,no,2476,1,0,2,5,8,579,1,-1,0,3,1 54,0,1,2,no,184,0,0,2,5,8,673,2,-1,0,3,1 42,4,2,2,no,0,1,1,2,5,8,562,2,-1,0,3,1 56,4,1,2,no,830,1,1,2,6,8,1201,1,-1,0,3,1 60,5,0,1,no,545,1,0,2,6,8,1030,1,-1,0,3,1 37,9,1,1,no,1,1,0,2,6,8,608,1,-1,0,3,1 28,7,2,1,no,5090,1,0,2,6,8,1297,3,-1,0,3,1 38,0,2,1,no,100,1,0,2,7,8,786,1,-1,0,3,1 30,1,1,1,no,309,1,0,2,7,8,1574,2,-1,0,3,1 29,4,1,2,no,199,1,1,2,7,8,1689,4,-1,0,3,1 46,1,2,2,no,460,1,0,2,7,8,1102,2,-1,0,3,1 31,9,2,2,no,703,1,0,2,8,8,943,2,-1,0,3,1 35,4,0,2,no,3837,1,0,2,8,8,1084,1,-1,0,3,1 32,1,2,0,no,611,1,0,2,8,8,541,3,-1,0,3,1 49,7,1,1,no,-8,1,0,2,8,8,1119,1,-1,0,3,1 41,0,1,1,no,55,1,0,2,8,8,1120,2,-1,0,3,1 49,0,0,1,no,168,1,1,2,8,8,513,1,-1,0,3,1 28,0,0,1,no,785,1,0,2,8,8,442,2,-1,0,3,1 43,4,2,2,no,2067,1,0,2,8,8,756,1,-1,0,3,1 43,4,0,2,no,388,1,0,2,8,8,2087,2,-1,0,3,1 43,1,1,0,no,-192,1,0,2,8,8,1120,2,-1,0,3,1 37,10,2,1,no,381,1,0,2,8,8,985,2,-1,0,3,1 35,1,2,1,no,40,1,0,2,9,8,617,4,-1,0,3,1 31,9,2,2,no,22,1,0,2,9,8,483,3,-1,0,3,1 43,1,2,1,no,3,1,0,2,9,8,929,3,-1,0,3,1 31,0,1,1,no,307,1,0,2,9,8,538,1,-1,0,3,1 28,1,2,1,no,759,1,0,2,9,8,710,1,-1,0,3,1 32,1,1,1,yes,-1,1,0,2,9,8,653,1,-1,0,3,1 60,9,1,0,no,65,1,0,2,9,8,1028,2,-1,0,3,1 26,1,2,1,no,82,1,0,2,9,8,654,1,-1,0,3,1 40,1,1,1,no,10,1,0,2,9,8,1692,2,-1,0,3,1 33,1,0,0,no,390,1,0,2,9,8,665,2,-1,0,3,1 32,4,2,2,no,311,0,0,2,12,8,757,2,-1,0,3,1 35,1,1,0,no,414,0,0,2,13,8,504,4,-1,0,3,1 33,4,0,1,no,5,1,0,2,13,8,1346,1,-1,0,3,1 38,4,1,2,no,119,1,0,2,13,8,568,4,-1,0,3,1 23,2,2,0,no,4,1,0,2,13,8,395,2,-1,0,3,1 60,1,1,0,no,1262,1,1,2,13,8,1015,1,-1,0,3,1 48,4,1,2,no,1949,1,0,2,13,8,683,2,-1,0,3,1 45,2,0,2,no,-395,1,0,2,13,8,470,1,-1,0,3,1 36,1,1,1,no,1165,1,0,2,13,8,1001,4,-1,0,3,1 52,4,1,3,no,2240,1,0,2,13,8,845,1,-1,0,3,1 35,1,0,0,no,300,1,0,2,13,8,945,2,-1,0,3,1 43,9,1,1,no,3285,1,0,2,13,8,1721,2,-1,0,3,1 52,3,1,1,no,3923,1,0,2,14,8,942,3,-1,0,3,1 53,9,0,0,no,1443,1,0,2,14,8,476,1,-1,0,3,1 48,1,0,0,no,24,1,0,2,14,8,832,1,-1,0,3,1 41,1,2,0,no,1618,1,0,2,14,8,1553,1,-1,0,3,1 39,4,0,2,no,517,1,1,2,14,8,1328,1,-1,0,3,1 59,5,1,1,no,1521,1,1,2,14,8,1125,2,-1,0,3,1 41,1,1,1,no,2823,1,0,2,14,8,858,1,-1,0,3,1 48,1,1,1,no,1405,1,0,2,14,8,629,2,-1,0,3,1 40,0,1,1,no,1535,1,0,2,14,8,704,1,-1,0,3,1 48,1,1,1,no,1596,1,0,2,14,8,760,1,-1,0,3,1 60,5,1,0,no,1542,1,0,2,14,8,930,1,-1,0,3,1 40,9,2,3,no,3652,1,0,2,14,8,1028,2,-1,0,3,1 57,9,1,2,no,-1,0,0,2,14,8,850,2,-1,0,3,1 51,1,1,1,no,7180,1,0,2,14,8,927,1,-1,0,3,1 41,1,0,1,no,5291,1,0,2,14,8,1423,3,-1,0,3,1 41,1,1,1,no,1384,1,0,2,15,8,1162,4,-1,0,3,1 52,1,0,0,no,-191,1,0,2,15,8,755,1,-1,0,3,1 59,1,1,0,no,320,1,0,2,15,8,695,1,-1,0,3,1 44,1,1,3,no,146,1,0,2,15,8,483,1,-1,0,3,1 49,11,1,0,no,341,1,1,2,15,8,520,2,-1,0,3,1 40,7,1,0,no,-9,1,0,2,15,8,920,2,-1,0,3,1 41,0,1,0,no,-306,1,0,2,15,8,500,1,-1,0,3,1 44,9,1,2,no,4580,1,0,2,15,8,911,2,-1,0,3,1 60,7,2,0,no,313,1,0,2,15,8,920,2,-1,0,3,1 29,4,1,2,no,10576,0,0,2,15,8,1224,2,-1,0,3,1 41,5,0,0,no,-233,1,0,2,15,8,1156,2,-1,0,3,1 41,1,1,0,no,2453,1,0,2,15,8,1052,1,-1,0,3,1 42,9,2,1,no,1364,1,0,2,15,8,1867,6,-1,0,3,1 36,7,2,3,no,281,1,0,2,16,8,515,1,-1,0,3,1 39,9,1,2,no,94,1,1,2,16,8,813,1,-1,0,3,1 31,6,2,2,no,144,1,0,2,16,8,676,1,-1,0,3,1 26,1,1,1,no,246,1,0,2,16,8,1143,3,-1,0,3,1 31,1,1,1,no,92,1,0,2,16,8,688,1,-1,0,3,1 29,0,2,1,no,163,1,1,2,16,8,803,1,-1,0,3,1 37,7,0,1,no,49,1,0,2,16,8,619,2,-1,0,3,1 33,1,2,1,no,-416,1,0,2,16,8,767,2,-1,0,3,1 24,9,2,1,no,409,1,0,2,16,8,912,1,-1,0,3,1 50,0,0,1,no,363,1,0,2,16,8,1340,1,-1,0,3,1 27,9,2,2,no,3706,1,0,2,16,8,897,1,-1,0,3,1 54,4,2,2,no,4393,1,0,2,16,8,1297,3,-1,0,3,1 30,9,2,1,no,863,1,0,2,16,8,1193,5,-1,0,3,1 34,1,1,0,no,695,1,1,2,16,8,1064,4,-1,0,3,1 37,4,2,1,no,792,1,0,2,16,8,1187,4,-1,0,3,1 41,0,2,1,no,1020,1,0,2,19,8,882,1,-1,0,3,1 47,1,1,1,no,863,1,0,2,19,8,943,2,-1,0,3,1 31,0,1,1,no,97,1,0,2,19,8,607,1,-1,0,3,1 31,1,1,1,no,754,1,0,2,19,8,1022,4,-1,0,3,1 43,7,0,1,no,1040,1,0,2,19,8,552,3,-1,0,3,1 37,4,0,2,no,122,1,0,2,19,8,1622,2,-1,0,3,1 30,9,2,2,no,880,1,1,2,19,8,967,6,-1,0,3,1 41,9,1,1,no,501,1,0,2,19,8,579,5,-1,0,3,1 36,1,1,1,no,4438,1,0,2,20,8,446,1,-1,0,3,1 35,1,0,1,no,0,1,0,2,20,8,1205,1,-1,0,3,1 29,7,2,1,no,271,1,0,2,20,8,1882,2,-1,0,3,1 35,1,1,0,no,102,1,0,2,20,8,1334,1,-1,0,3,1 31,1,1,1,no,2,1,0,2,20,8,182,2,-1,0,3,1 35,7,1,1,no,4170,1,0,2,20,8,1777,1,-1,0,3,1 34,9,2,1,no,85,1,0,2,20,8,1182,4,-1,0,3,1 31,0,2,1,no,431,1,1,2,20,8,1045,1,-1,0,3,1 44,9,0,1,no,982,1,0,2,20,8,650,3,-1,0,3,1 36,1,1,0,no,408,1,1,2,20,8,1063,2,-1,0,3,1 35,1,1,3,no,4822,1,0,2,20,8,843,2,-1,0,3,1 41,1,1,0,no,1250,1,0,2,20,8,1392,2,-1,0,3,1 31,1,2,0,no,216,1,0,2,21,8,565,1,-1,0,3,1 34,1,1,1,no,1207,1,0,2,21,8,905,1,-1,0,3,1 32,9,1,1,no,791,1,0,2,21,8,783,2,-1,0,3,1 36,4,1,2,no,849,1,1,2,21,8,958,2,-1,0,3,1 30,1,2,1,yes,239,1,0,2,21,8,412,1,-1,0,3,1 37,9,2,1,no,1211,1,0,2,21,8,795,1,-1,0,3,1 36,1,1,0,no,599,1,0,2,21,8,649,2,-1,0,3,1 31,4,2,2,no,825,1,0,2,21,8,506,2,-1,0,3,1 27,9,1,1,no,2183,1,1,2,21,8,857,1,-1,0,3,1 34,5,1,0,no,4499,0,0,2,21,8,1681,1,-1,0,3,1 30,9,2,2,no,1289,1,0,2,21,8,1697,2,-1,0,3,1 32,9,2,2,no,4665,1,0,2,21,8,860,2,-1,0,3,1 37,9,2,1,no,3326,1,0,2,21,8,799,1,-1,0,3,1 32,1,1,1,no,783,1,0,2,21,8,923,3,-1,0,3,1 33,1,1,1,no,0,1,0,2,21,8,521,3,-1,0,3,1 35,1,1,0,no,994,1,0,2,23,8,1349,8,-1,0,3,1 35,6,0,2,no,1354,1,0,2,23,8,736,2,-1,0,3,1 30,1,2,1,no,239,1,0,2,23,8,785,1,-1,0,3,1 45,1,0,0,no,-311,1,0,2,23,8,1030,1,-1,0,3,1 45,0,1,1,no,149,1,0,2,23,8,893,3,-1,0,3,1 33,1,1,1,no,1464,1,0,2,23,8,588,3,-1,0,3,1 44,4,0,2,no,5773,0,0,2,23,8,597,6,-1,0,3,1 40,1,1,0,no,278,1,0,2,23,8,1015,3,-1,0,3,1 31,1,1,1,no,2910,1,0,2,23,8,1392,2,-1,0,3,1 30,9,1,2,no,541,1,0,2,26,8,414,2,-1,0,3,1 44,9,0,1,no,1262,1,0,2,26,8,788,11,-1,0,3,1 34,7,1,1,no,-538,1,0,2,26,8,682,1,-1,0,3,1 37,1,1,1,no,125,0,0,2,26,8,679,2,-1,0,3,1 28,6,2,2,no,0,1,0,2,26,8,1560,2,-1,0,3,1 32,0,2,1,no,620,1,0,2,26,8,1234,3,-1,0,3,1 37,1,0,1,no,316,1,0,2,26,8,642,2,-1,0,3,1 34,4,1,2,no,2287,1,0,2,26,8,895,4,-1,0,3,1 38,0,0,1,no,198,1,0,2,26,8,431,2,-1,0,3,1 28,0,2,1,no,460,1,0,2,26,8,741,6,-1,0,3,1 31,4,0,2,no,1145,1,0,2,26,8,1272,4,-1,0,3,1 59,7,1,1,no,-22,1,0,2,26,8,748,3,-1,0,3,1 24,1,1,1,no,685,1,0,2,26,8,896,3,-1,0,3,1 26,7,2,1,no,901,1,0,2,26,8,764,2,-1,0,3,1 59,5,2,1,no,351,1,0,2,27,8,1063,5,-1,0,3,1 30,2,2,0,no,0,1,1,2,27,8,1051,1,-1,0,3,1 37,0,1,1,no,274,1,0,2,27,8,731,3,-1,0,3,1 30,0,2,1,no,-213,1,1,2,27,8,751,1,-1,0,3,1 25,1,1,1,no,97,1,0,2,27,8,709,1,-1,0,3,1 44,7,0,1,no,51,1,1,2,27,8,3094,2,-1,0,3,1 29,4,1,2,no,314,0,0,2,28,8,938,2,-1,0,3,1 26,4,2,2,no,6840,1,0,2,28,8,1560,2,-1,0,3,1 26,9,2,1,no,668,1,0,2,28,8,576,3,-1,0,3,1 27,7,2,1,no,54,1,0,2,28,8,543,2,-1,0,3,1 25,6,2,1,no,1242,1,0,2,28,8,615,2,-1,0,3,1 28,4,1,1,no,292,1,0,2,28,8,507,1,-1,0,3,1 23,7,2,1,no,665,1,0,2,28,8,1183,1,-1,0,3,1 34,4,2,2,no,1058,1,0,2,28,8,864,2,-1,0,3,1 28,8,2,1,no,949,1,0,2,28,8,1730,1,-1,0,3,1 51,2,1,3,no,606,1,0,2,28,8,560,9,-1,0,3,1 35,1,1,1,no,0,1,1,2,29,8,543,1,-1,0,3,1 35,0,1,1,no,404,1,0,2,29,8,539,1,-1,0,3,1 56,1,1,1,no,249,1,0,2,29,8,791,2,-1,0,3,1 45,10,0,1,no,3354,1,0,2,29,8,746,1,-1,0,3,1 60,1,0,1,no,0,1,0,2,29,8,1207,1,-1,0,3,1 56,9,1,1,no,589,1,0,2,29,8,535,2,-1,0,3,1 27,4,2,2,no,12956,1,0,2,29,8,789,3,-1,0,3,1 30,0,1,1,no,873,1,0,2,29,8,792,3,-1,0,3,1 45,1,0,0,no,594,1,0,2,29,8,833,2,-1,0,3,1 29,1,2,1,no,-20,1,0,2,29,8,814,2,-1,0,3,1 38,4,0,2,no,4692,1,0,2,29,8,1363,2,-1,0,3,1 30,1,1,1,no,486,1,0,2,29,8,1109,3,-1,0,3,1 44,1,2,0,no,1593,1,0,2,29,8,828,3,-1,0,3,1 36,4,2,2,no,7606,1,0,2,30,8,917,1,-1,0,3,1 38,9,1,1,no,226,1,0,2,30,8,762,3,-1,0,3,1 36,1,2,1,no,366,1,1,2,30,8,1133,2,-1,0,3,1 37,4,0,1,no,16,1,0,2,30,8,638,1,-1,0,3,1 35,4,2,2,no,565,1,0,2,30,8,763,1,-1,0,3,1 38,0,0,1,no,19,1,0,2,30,8,604,2,-1,0,3,1 40,7,1,0,no,1082,1,0,2,30,8,854,2,-1,0,3,1 42,10,0,1,no,1713,1,0,2,30,8,855,2,-1,0,3,1 46,4,0,2,no,14481,1,0,2,30,8,1269,1,-1,0,3,1 35,9,0,1,no,5724,1,0,2,30,8,691,10,-1,0,3,1 34,9,0,1,no,1451,1,0,2,30,8,1097,15,-1,0,3,1 32,7,1,1,no,-34,1,0,2,30,8,1236,2,-1,0,3,1 50,10,1,1,no,3674,1,0,2,30,8,886,3,-1,0,3,1 35,1,1,1,no,698,1,1,2,30,8,1343,3,-1,0,3,1 36,9,2,1,no,4136,1,0,2,30,8,812,3,-1,0,3,1 53,1,0,0,no,2656,1,0,2,30,8,1980,2,-1,0,3,1 46,4,2,2,no,2904,1,0,2,30,8,984,2,-1,0,3,1 26,4,2,2,no,1004,1,0,2,30,8,228,2,-1,0,3,1 45,9,2,1,no,410,1,0,2,30,8,891,4,-1,0,3,1 31,10,0,0,no,0,1,0,2,30,8,745,12,-1,0,3,1 29,4,2,2,no,0,1,0,2,30,8,539,12,-1,0,3,1 53,2,2,1,no,62,1,0,2,30,8,1044,5,-1,0,3,1 27,1,1,0,no,416,1,0,2,30,8,1193,2,-1,0,3,1 39,4,1,2,no,2763,1,0,2,2,6,526,2,-1,0,3,1 36,7,1,1,no,2984,1,0,2,2,6,394,1,-1,0,3,1 41,1,1,0,no,143,1,1,2,2,6,659,2,-1,0,3,1 36,7,1,0,no,0,1,0,2,2,6,1036,1,-1,0,3,1 32,9,1,1,no,696,1,0,2,2,6,815,1,-1,0,3,1 35,7,1,1,no,152,1,0,2,2,6,563,1,-1,0,3,1 36,4,2,2,no,-78,1,0,2,2,6,1068,1,-1,0,3,1 35,4,1,2,no,867,1,0,2,2,6,230,3,-1,0,3,1 30,1,2,1,no,953,1,0,2,2,6,747,2,-1,0,3,1 42,1,1,1,no,443,1,0,2,2,6,671,5,-1,0,3,1 42,9,2,1,yes,-1129,1,0,2,2,6,555,5,-1,0,3,1 32,1,1,0,no,415,1,0,2,2,6,777,4,-1,0,3,1 29,1,2,1,no,260,1,0,2,2,6,707,14,-1,0,3,1 44,4,0,2,yes,-411,1,0,2,2,6,432,4,-1,0,3,1 32,9,2,1,no,1279,1,0,2,3,6,1173,3,-1,0,3,1 31,10,1,1,no,86,1,1,2,3,6,963,2,-1,0,3,1 40,6,1,1,yes,-754,1,1,2,3,6,941,3,-1,0,3,1 43,2,1,1,no,0,0,0,2,3,6,1025,7,-1,0,3,1 27,4,2,1,no,178,1,0,2,3,6,732,2,-1,0,3,1 37,7,2,1,no,278,1,0,2,3,6,1045,3,-1,0,3,1 23,7,2,1,no,425,1,0,2,3,6,768,1,-1,0,3,1 25,4,1,0,no,168,1,0,2,3,6,801,1,-1,0,3,1 26,1,2,1,yes,-36,1,0,2,3,6,482,2,-1,0,3,1 47,1,0,0,no,126,1,0,2,3,6,2456,2,-1,0,3,1 35,0,1,1,no,-87,1,0,2,3,6,1363,7,-1,0,3,1 32,9,2,2,no,6281,1,0,2,3,6,1336,4,-1,0,3,1 45,4,0,1,no,644,1,0,2,4,6,633,1,-1,0,3,1 30,0,0,1,no,377,1,0,2,4,6,524,15,-1,0,3,1 29,7,2,1,no,-271,1,0,2,4,6,578,1,-1,0,3,1 38,1,1,0,no,5839,1,0,2,4,6,984,2,-1,0,3,1 23,1,2,1,no,492,1,0,2,4,6,638,1,-1,0,3,1 27,4,2,2,no,99,1,0,2,4,6,767,2,-1,0,3,1 27,4,2,2,yes,-713,1,0,2,4,6,525,2,-1,0,3,1 29,7,2,1,no,32,1,0,2,4,6,563,1,-1,0,3,1 30,6,1,2,no,543,1,1,2,4,6,1449,6,-1,0,3,1 36,1,1,0,no,403,1,0,2,4,6,920,1,-1,0,3,1 35,0,0,1,no,437,1,0,2,4,6,908,1,-1,0,3,1 36,1,1,1,no,1772,0,0,2,5,6,208,1,-1,0,3,1 59,5,0,0,no,-88,1,0,2,5,6,910,2,-1,0,3,1 47,2,1,1,no,0,0,0,2,5,6,1446,1,-1,0,3,1 58,2,1,2,no,853,0,0,2,5,6,1149,2,-1,0,3,1 37,4,2,2,no,240,0,0,2,5,6,1123,1,-1,0,3,1 46,0,2,1,no,849,1,0,2,5,6,508,2,-1,0,3,1 43,1,2,1,no,318,1,0,2,5,6,736,2,-1,0,3,1 27,4,2,2,no,703,1,0,2,5,6,590,3,-1,0,3,1 38,7,0,1,no,61,0,0,2,5,6,636,1,-1,0,3,1 53,3,1,0,no,104,0,1,2,5,6,701,1,-1,0,3,1 47,1,1,1,no,1996,0,0,2,5,6,761,2,-1,0,3,1 56,3,0,0,yes,1238,0,0,2,5,6,1558,1,-1,0,3,1 41,3,1,1,no,879,0,0,2,5,6,1053,2,-1,0,3,1 29,6,1,1,no,425,1,0,2,5,6,562,15,-1,0,3,1 50,9,1,1,no,416,1,0,2,5,6,494,5,-1,0,3,1 45,1,0,0,no,844,0,0,2,5,6,1018,3,-1,0,3,1 35,1,1,1,no,860,1,0,2,5,6,884,4,-1,0,3,1 26,9,2,1,no,639,1,0,2,6,6,709,5,-1,0,3,1 26,0,1,1,no,710,1,1,2,6,6,1276,2,-1,0,3,1 37,1,1,3,no,1508,0,1,2,6,6,381,2,-1,0,3,1 30,0,2,1,no,223,1,0,2,6,6,862,11,-1,0,3,1 38,4,2,2,no,619,1,0,2,6,6,460,4,-1,0,3,1 37,7,0,1,no,319,1,0,2,6,6,467,3,-1,0,3,1 53,9,1,1,no,480,0,0,2,6,6,648,2,-1,0,3,1 29,1,2,0,no,213,1,0,2,6,6,2653,3,-1,0,3,1 35,4,2,2,no,68,0,0,2,6,6,1085,1,-1,0,3,1 28,1,1,1,no,2788,1,0,2,6,6,369,2,-1,0,3,1 55,1,1,3,no,7561,0,0,2,6,6,685,3,-1,0,3,1 49,4,0,1,no,-46,1,0,2,9,6,1055,10,-1,0,3,1 50,7,1,1,no,1595,0,0,2,9,6,882,1,-1,0,3,1 43,4,2,2,no,1046,0,0,2,9,6,901,1,-1,0,3,1 58,6,0,2,no,1382,0,0,2,9,6,700,1,-1,0,3,1 38,9,1,0,no,217,1,0,2,9,6,491,6,-1,0,3,1 48,1,0,0,no,183,0,0,2,9,6,940,1,-1,0,3,1 59,4,1,1,no,1321,0,0,2,9,6,3881,3,-1,0,3,1 58,1,1,1,no,25,0,0,2,9,6,486,3,-1,0,3,1 36,6,2,2,no,4844,0,0,2,9,6,1137,3,-1,0,3,1 42,2,1,2,no,5345,0,0,2,11,6,878,3,-1,0,3,1 41,0,2,1,no,812,0,1,2,11,6,583,4,-1,0,3,1 42,0,1,1,no,105,0,0,2,11,6,1159,4,-1,0,3,1 39,4,1,2,no,-190,0,1,2,11,6,893,8,-1,0,3,1 57,3,1,0,no,625,0,1,2,11,6,867,1,-1,0,3,1 52,5,1,3,no,293,0,0,2,11,6,706,1,-1,0,3,1 59,5,1,3,no,1033,0,0,2,11,6,1199,1,-1,0,3,1 56,1,1,1,no,1210,0,0,2,11,6,935,1,-1,0,3,1 49,4,1,2,no,64,0,0,2,11,6,586,1,-1,0,3,1 35,1,0,3,no,164,0,0,2,11,6,967,2,-1,0,3,1 36,10,1,1,no,1766,0,0,2,11,6,595,2,-1,0,3,1 40,0,2,2,no,1248,0,0,2,11,6,1290,2,-1,0,3,1 55,1,1,1,no,0,0,0,2,12,6,614,2,-1,0,3,1 58,0,1,1,no,826,0,0,2,12,6,885,1,-1,0,3,1 39,6,1,2,no,2630,0,0,2,12,6,651,5,-1,0,3,1 47,3,2,2,no,3727,0,0,2,12,6,993,6,-1,0,3,1 40,4,1,1,no,406,1,0,2,12,6,577,2,-1,0,3,1 35,1,1,2,no,2999,0,0,2,12,6,1141,6,-1,0,3,1 48,1,1,1,no,1000,1,0,2,12,6,1268,2,-1,0,3,1 48,4,0,2,no,503,0,1,2,12,6,1243,7,-1,0,3,1 24,1,2,1,yes,-203,1,0,2,12,6,465,3,-1,0,3,1 49,1,1,0,no,361,1,0,2,16,6,513,2,-1,0,3,1 36,1,1,1,no,274,1,0,2,16,6,686,11,-1,0,3,1 52,5,0,2,no,3687,1,1,2,16,6,741,4,-1,0,3,1 42,2,2,3,no,323,0,0,2,16,6,617,1,-1,0,3,1 36,7,1,1,no,981,0,0,2,16,6,1093,1,-1,0,3,1 36,1,0,1,no,638,1,0,2,16,6,1395,2,-1,0,3,1 48,10,2,2,no,3229,0,0,2,16,6,1089,1,-1,0,3,1 51,4,2,1,no,1610,1,0,2,16,6,1248,3,-1,0,3,1 51,1,1,1,no,33,0,1,2,16,6,721,3,-1,0,3,1 46,9,1,1,no,536,1,0,2,16,6,750,2,-1,0,3,1 37,4,1,2,no,0,0,0,2,16,6,637,6,-1,0,3,1 36,2,1,2,no,3057,0,0,2,16,6,2769,4,-1,0,3,1 47,4,1,1,no,2892,0,0,2,17,6,556,1,-1,0,3,1 40,9,0,2,no,0,0,0,2,17,6,522,1,-1,0,3,1 38,8,2,2,no,3316,0,0,2,17,6,1345,3,-1,0,3,1 40,4,1,1,no,106,0,0,2,17,6,676,2,-1,0,3,1 39,1,1,0,no,2303,1,0,2,17,6,775,3,-1,0,3,1 60,4,1,2,no,3301,0,0,2,17,6,2621,3,-1,0,3,1 39,6,2,2,no,426,0,0,2,18,6,1029,1,-1,0,3,1 51,7,0,1,no,9,0,0,2,18,6,371,2,-1,0,3,1 27,0,1,1,no,-97,1,0,2,18,6,1528,2,-1,0,3,1 41,4,2,2,no,11008,0,0,2,18,6,1540,1,-1,0,3,1 42,1,2,0,no,4930,0,0,2,18,6,973,1,-1,0,3,1 41,2,1,1,no,707,0,0,2,18,6,707,1,-1,0,3,1 33,9,1,1,no,4,1,1,2,18,6,625,2,-1,0,3,1 46,0,1,1,no,126,0,0,2,18,6,1255,1,-1,0,3,1 31,1,1,1,no,671,1,0,2,18,6,704,4,-1,0,3,1 41,1,1,0,no,219,0,1,2,18,6,1574,3,-1,0,3,1 41,1,0,1,no,3622,0,0,2,19,6,1135,4,-1,0,3,1 55,5,0,1,no,1580,0,1,2,19,6,1007,1,-1,0,3,1 53,9,1,1,no,1319,0,0,2,19,6,1318,3,-1,0,3,1 55,3,1,0,no,2152,0,0,2,19,6,922,2,-1,0,3,1 39,7,1,1,no,2843,1,0,2,19,6,585,1,-1,0,3,1 40,2,1,2,yes,0,0,0,2,19,6,952,4,-1,0,3,1 59,5,1,0,no,1115,0,0,2,19,6,834,1,-1,0,3,1 42,1,2,1,no,625,0,0,2,19,6,651,3,-1,0,3,1 41,1,1,0,no,246,0,0,2,19,6,683,3,-1,0,3,1 27,1,1,1,no,61,1,0,2,19,6,1012,1,-1,0,3,1 25,1,2,1,no,-191,1,0,2,19,6,958,7,-1,0,3,1 43,10,1,0,no,519,0,0,2,20,6,973,2,-1,0,3,1 44,7,1,1,no,522,0,1,2,20,6,911,1,-1,0,3,1 26,1,2,1,no,-46,1,0,2,20,6,382,7,-1,0,3,1 52,5,1,2,no,2269,0,0,2,20,6,1210,1,-1,0,3,1 54,7,1,3,no,386,0,1,2,20,6,838,1,-1,0,3,1 56,4,1,2,no,0,0,1,2,20,6,422,1,-1,0,3,1 57,4,1,2,no,9103,0,1,2,20,6,1098,2,-1,0,3,1 29,1,2,1,no,908,1,0,2,20,6,1663,1,-1,0,3,1 47,9,1,2,no,1694,0,1,2,20,6,560,3,-1,0,3,1 57,4,1,2,no,-92,1,1,2,20,6,1617,1,-1,0,3,1 40,1,1,1,no,188,1,1,2,20,6,570,2,-1,0,3,1 46,7,0,1,no,0,0,0,2,20,6,1023,3,-1,0,3,1 43,1,2,1,no,740,1,0,2,20,6,585,2,-1,0,3,1 45,4,1,2,no,655,0,0,2,20,6,693,3,-1,0,3,1 24,4,1,1,no,867,1,0,2,20,6,546,10,-1,0,3,1 40,1,1,1,no,1009,1,1,2,20,6,1036,5,-1,0,3,1 38,7,0,3,no,6360,0,0,2,20,6,1409,2,-1,0,3,1 36,10,1,2,no,4145,0,0,2,20,6,988,4,-1,0,3,1 27,1,2,1,yes,-27,0,0,2,23,6,498,2,-1,0,3,1 26,0,2,0,no,-122,1,0,2,23,6,670,3,-1,0,3,1 43,4,1,2,no,149,1,0,2,23,6,662,2,-1,0,3,1 25,1,2,0,no,52,1,0,2,24,6,528,2,-1,0,3,1 27,1,1,2,no,335,1,0,2,26,6,519,3,-1,0,3,1 30,9,1,2,no,526,1,1,2,26,6,677,3,-1,0,3,1 28,0,2,1,no,-7,0,1,2,26,6,3183,2,-1,0,3,1 27,1,1,0,no,-413,0,1,2,27,6,422,6,-1,0,3,1 32,0,0,1,no,-241,0,1,2,27,6,856,2,-1,0,3,1 41,1,2,1,no,75,1,0,2,1,5,479,9,-1,0,3,1 37,9,2,2,no,0,0,0,2,1,5,1135,2,-1,0,3,1 27,9,2,1,no,158,0,0,2,1,5,650,2,-1,0,3,1 29,7,2,1,no,84,0,1,2,3,5,587,3,-1,0,3,1 33,7,1,1,no,114,0,1,2,3,5,676,1,-1,0,3,1 36,9,1,0,no,0,0,0,2,4,5,584,1,-1,0,3,1 28,1,2,1,no,-127,1,0,0,4,5,1044,3,-1,0,3,1 55,1,1,1,no,49,1,0,0,7,5,494,4,-1,0,3,1 51,9,1,1,no,216,0,0,0,7,5,615,2,-1,0,3,1 39,9,2,1,no,506,1,0,0,7,5,718,2,-1,0,3,1 42,1,0,0,no,213,1,0,0,7,5,434,3,-1,0,3,1 33,7,2,1,no,170,0,0,0,7,5,782,1,-1,0,3,1 36,2,0,2,no,776,0,1,0,7,5,722,3,-1,0,3,1 45,9,1,1,no,879,0,0,0,7,5,621,2,-1,0,3,1 37,6,1,1,no,1633,0,0,0,7,5,629,2,-1,0,3,1 46,10,1,1,no,926,0,0,0,7,5,385,5,-1,0,3,1 32,6,2,2,no,2559,0,0,0,7,5,889,1,-1,0,3,1 28,9,2,1,no,909,0,1,0,7,5,532,1,-1,0,3,1 45,0,0,1,no,-176,0,1,0,7,5,574,1,-1,0,3,1 41,1,2,1,yes,1085,1,1,0,7,5,599,1,-1,0,3,1 59,1,0,0,no,1265,0,1,0,7,5,849,1,-1,0,3,1 47,9,2,1,no,-239,1,1,0,7,5,973,3,-1,0,3,1 31,1,2,1,no,726,0,0,0,7,5,719,2,-1,0,3,1 31,4,2,2,no,296,0,1,0,8,5,509,4,-1,0,3,1 33,2,1,1,no,-627,1,1,0,8,5,740,1,-1,0,3,1 52,2,1,2,yes,-468,0,0,0,8,5,534,1,-1,0,3,1 26,6,2,1,no,551,0,0,0,8,5,531,1,-1,0,3,1 25,9,2,1,no,0,0,1,0,8,5,634,1,-1,0,3,1 25,1,1,1,no,101,0,1,0,8,5,460,3,-1,0,3,1 28,1,2,1,no,517,0,1,0,8,5,681,1,-1,0,3,1 33,2,0,2,no,37,0,1,0,8,5,1082,1,-1,0,3,1 45,4,0,2,no,854,0,1,0,8,5,730,1,-1,0,3,1 36,7,1,1,no,30,0,1,0,8,5,716,5,-1,0,3,1 26,1,2,1,yes,81,0,1,0,8,5,803,2,-1,0,3,1 38,9,1,1,no,0,0,0,0,8,5,635,1,-1,0,3,1 56,1,2,1,no,663,0,1,0,8,5,819,3,-1,0,3,1 26,0,2,1,no,102,0,1,0,8,5,658,2,-1,0,3,1 26,9,1,1,no,370,0,1,0,8,5,1061,4,-1,0,3,1 25,1,2,1,no,189,0,1,0,8,5,687,2,-1,0,3,1 26,10,2,1,no,111,0,0,0,9,5,537,1,-1,0,3,1 40,1,1,0,no,665,1,0,0,9,5,781,1,-1,0,3,1 37,0,1,0,no,214,1,0,0,9,5,645,1,-1,0,3,1 43,1,2,0,no,0,0,0,0,9,5,1242,1,-1,0,3,1 41,7,0,2,no,1583,1,0,0,9,5,662,1,-1,0,3,1 49,1,1,1,no,0,0,1,0,9,5,625,1,-1,0,3,1 29,0,0,2,yes,0,0,1,0,9,5,426,3,-1,0,3,1 39,7,0,1,no,687,1,0,0,9,5,869,1,-1,0,3,1 38,4,1,2,yes,327,1,1,0,9,5,556,5,-1,0,3,1 26,9,1,1,no,-119,0,1,0,9,5,359,5,-1,0,3,1 32,4,1,2,no,794,1,1,0,9,5,1014,1,-1,0,3,1 33,1,1,0,no,-191,1,1,0,9,5,1290,4,-1,0,3,1 31,4,2,2,no,409,0,1,0,9,5,564,5,-1,0,3,1 49,1,1,0,no,3,0,1,0,9,5,1439,6,-1,0,3,1 27,10,2,0,no,9,0,1,0,9,5,919,2,-1,0,3,1 30,1,2,1,no,155,1,1,0,9,5,1426,3,-1,0,3,1 35,1,1,1,no,1330,0,0,0,10,5,901,1,-1,0,3,1 26,1,2,1,no,-29,1,0,0,10,5,998,4,-1,0,3,1 45,2,1,3,no,3133,1,1,0,10,5,804,1,-1,0,3,1 34,0,1,1,no,-33,1,1,0,10,5,961,1,-1,0,3,1 28,0,2,1,no,-170,0,1,0,10,5,720,2,-1,0,3,1 25,1,2,1,no,1243,0,1,0,10,5,1341,7,-1,0,3,1 30,1,2,1,no,1598,1,0,0,10,5,634,1,-1,0,3,1 30,7,1,1,no,5,0,1,0,10,5,829,5,-1,0,3,1 47,7,1,1,no,367,1,0,0,10,5,476,4,-1,0,3,1 57,0,0,1,no,658,0,0,0,10,5,724,1,-1,0,3,1 38,4,1,2,no,1722,1,0,0,10,5,500,1,-1,0,3,1 54,9,1,1,no,-315,0,1,0,10,5,2029,1,-1,0,3,1 25,1,2,0,no,2483,0,0,0,10,5,1499,3,-1,0,3,1 30,1,1,1,no,0,1,1,0,10,5,1399,1,-1,0,3,1 26,4,2,2,no,45,0,0,0,10,5,1187,3,-1,0,3,1 31,8,1,0,no,46,1,0,0,10,5,487,1,-1,0,3,1 29,1,2,1,no,482,0,1,0,10,5,1097,3,-1,0,3,1 49,1,1,1,no,3728,1,0,0,11,5,1060,2,-1,0,3,1 60,5,0,1,no,1099,0,0,0,11,5,764,2,-1,0,3,1 60,4,1,2,no,2222,1,0,0,11,5,1120,2,-1,0,3,1 29,4,2,2,no,983,1,0,0,11,5,963,2,-1,0,3,1 55,4,0,1,no,204,1,0,0,11,5,1973,2,-1,0,3,1 44,0,1,1,no,1074,1,0,0,11,5,911,2,-1,0,3,1 47,9,1,1,no,3466,1,0,0,11,5,853,2,-1,0,3,1 35,0,1,1,no,-203,1,0,0,11,5,1649,2,-1,0,3,1 35,2,1,1,no,2436,0,1,0,14,5,1397,3,-1,0,3,1 38,9,1,1,no,976,1,0,0,14,5,766,3,-1,0,3,1 34,2,1,2,no,146,1,0,0,14,5,788,2,-1,0,3,1 50,9,1,1,no,156,0,1,0,14,5,1130,3,-1,0,3,1 43,0,1,2,no,189,1,0,0,14,5,1062,4,-1,0,3,1 44,0,0,1,no,163,1,0,0,14,5,1669,2,-1,0,3,1 51,7,0,1,no,242,0,1,0,14,5,1336,4,-1,0,3,1 31,9,1,1,no,1173,1,0,0,14,5,444,3,-1,0,3,1 39,9,1,2,no,0,1,0,0,14,5,894,7,-1,0,3,1 59,4,1,2,no,296,0,1,0,15,5,805,3,-1,0,3,1 25,7,2,1,no,333,0,1,0,15,5,1056,2,-1,0,3,1 58,4,1,2,no,3864,1,0,0,15,5,815,4,-1,0,3,1 30,2,1,0,no,2686,1,0,0,15,5,808,3,-1,0,3,1 34,1,2,0,no,-109,1,0,0,15,5,860,4,-1,0,3,1 51,7,0,1,no,65,0,1,0,15,5,881,1,-1,0,3,1 60,5,1,1,no,597,0,0,0,15,5,981,1,-1,0,3,1 51,2,1,1,no,-799,0,1,0,15,5,1001,3,-1,0,3,1 37,4,0,2,no,1775,0,0,0,15,5,514,5,-1,0,3,1 30,1,2,0,no,0,0,0,0,15,5,716,3,-1,0,3,1 46,2,0,1,no,-563,0,1,0,15,5,769,4,-1,0,3,1 29,8,2,2,no,5,0,0,0,15,5,889,5,-1,0,3,1 27,2,1,2,no,139,0,0,0,15,5,448,5,-1,0,3,1 28,4,2,2,no,0,1,0,0,16,5,767,5,-1,0,3,1 34,1,2,1,no,855,1,0,0,16,5,578,1,-1,0,3,1 29,1,2,3,no,694,1,1,0,16,5,1806,4,-1,0,3,1 27,4,2,2,no,151,1,0,0,16,5,496,1,-1,0,3,1 26,9,2,1,no,354,0,0,0,16,5,586,1,-1,0,3,1 36,4,1,2,no,673,1,0,0,16,5,689,1,-1,0,3,1 44,1,1,0,no,1265,1,0,0,16,5,555,1,-1,0,3,1 39,1,1,2,no,597,0,0,0,16,5,686,6,-1,0,3,1 30,9,1,1,no,-393,1,1,0,16,5,435,1,-1,0,3,1 31,10,0,2,no,256,1,0,0,16,5,873,2,-1,0,3,1 56,1,1,0,no,0,1,0,0,16,5,1222,1,-1,0,3,1 29,1,1,1,no,2870,1,0,0,16,5,988,1,-1,0,3,1 43,1,1,1,no,184,0,1,0,16,5,1019,11,-1,0,3,1 32,1,1,1,no,1905,1,0,0,16,5,709,2,-1,0,3,1 30,0,2,1,no,-36,0,0,0,16,5,561,9,-1,0,3,1 40,1,1,1,no,-839,1,1,0,16,5,1018,2,-1,0,3,1 42,7,1,1,no,3465,1,1,0,16,5,1039,2,-1,0,3,1 27,8,1,1,no,119,1,0,0,16,5,781,2,-1,0,3,1 35,4,1,2,no,36,0,1,0,16,5,1656,5,-1,0,3,1 35,3,1,0,no,113,1,0,0,17,5,923,2,-1,0,3,1 32,1,2,1,no,118,1,0,0,17,5,1275,1,-1,0,3,1 39,9,1,1,no,251,1,0,0,17,5,342,2,-1,0,3,1 31,1,0,1,no,-277,1,0,1,17,5,1008,2,-1,0,3,1 26,7,2,1,no,-189,0,1,0,17,5,538,2,-1,0,3,1 42,4,1,2,no,4,1,0,0,17,5,795,1,-1,0,3,1 27,7,2,1,no,1820,1,0,0,17,5,1027,1,-1,0,3,1 33,6,0,1,no,1904,1,0,0,17,5,1584,2,-1,0,3,1 51,1,1,0,no,201,1,0,0,17,5,582,2,-1,0,3,1 33,0,1,1,no,0,1,0,1,17,5,1448,7,-1,0,3,1 30,9,2,2,no,178,0,0,0,17,5,656,3,-1,0,3,1 52,3,1,0,no,3798,1,0,0,17,5,1208,1,-1,0,3,1 42,1,0,0,no,301,1,0,0,17,5,1175,2,-1,0,3,1 31,4,0,2,no,294,1,0,0,17,5,536,5,-1,0,3,1 27,7,1,1,no,0,1,0,0,17,5,991,5,-1,0,3,1 44,1,2,1,no,292,0,1,0,17,5,1153,4,-1,0,3,1 40,2,0,1,no,2998,1,0,0,18,5,623,3,-1,0,3,1 46,3,0,0,no,2084,1,0,0,18,5,1081,2,-1,0,3,1 32,7,0,1,no,454,1,0,0,18,5,978,1,-1,0,3,1 56,9,0,1,no,179,0,0,0,18,5,536,1,-1,0,3,1 33,1,1,1,no,661,1,0,0,18,5,968,1,-1,0,3,1 33,1,1,1,no,89,1,0,0,18,5,278,2,-1,0,3,1 32,6,1,0,no,102,1,0,0,18,5,560,1,-1,0,3,1 41,1,0,0,no,138,1,0,0,18,5,640,3,-1,0,3,1 31,0,2,1,no,622,0,0,0,18,5,420,1,-1,0,3,1 30,0,0,1,no,10,1,0,0,18,5,658,2,-1,0,3,1 34,0,0,1,no,-251,0,0,0,18,5,641,1,-1,0,3,1 43,2,1,1,no,104,1,0,0,18,5,635,3,-1,0,3,1 28,1,1,1,no,173,0,0,0,18,5,588,7,-1,0,3,1 32,0,0,1,no,-46,0,1,0,18,5,1373,4,-1,0,3,1 31,1,0,1,no,0,1,1,0,18,5,913,4,-1,0,3,1 35,2,2,2,no,145,1,0,0,18,5,799,2,-1,0,3,1 31,4,1,2,no,325,1,0,0,18,5,1139,2,-1,0,3,1 25,7,2,1,no,2311,0,1,0,21,5,1105,2,-1,0,3,1 56,5,0,2,no,158,0,1,0,21,5,854,1,-1,0,3,1 47,9,1,1,no,1981,1,0,0,21,5,919,2,-1,0,3,1 39,4,1,2,no,1423,1,0,1,21,5,733,3,-1,0,3,1 51,4,1,2,no,2455,1,0,0,21,5,553,1,-1,0,3,1 54,9,0,1,no,859,0,1,0,21,5,710,1,-1,0,3,1 58,5,1,3,no,9004,0,0,0,21,5,891,4,-1,0,3,1 42,10,1,3,no,970,1,0,1,21,5,691,10,-1,0,3,1 56,7,1,0,no,486,0,1,1,21,5,1877,1,-1,0,3,1 55,3,1,1,no,127,1,0,1,21,5,1130,8,-1,0,3,1 26,1,2,0,no,409,1,0,0,21,5,577,2,-1,0,3,1 40,0,2,1,no,664,0,0,0,21,5,1342,2,-1,0,3,1 26,9,2,1,no,223,0,1,0,21,5,1002,3,-1,0,3,1 35,4,2,2,no,-324,0,1,0,21,5,985,11,-1,0,3,1 34,7,2,1,no,-30,0,0,0,21,5,1360,3,-1,0,3,1 41,4,2,2,no,1253,0,0,0,21,5,1134,5,-1,0,3,1 44,4,1,0,no,4758,1,0,0,22,5,712,6,-1,0,3,1 33,4,2,2,no,30,1,0,0,22,5,1077,1,-1,0,3,1 36,4,0,2,no,81,0,1,0,22,5,560,1,-1,0,3,1 38,9,1,1,no,0,1,0,0,22,5,1545,1,-1,0,3,1 35,9,1,2,no,328,1,0,0,22,5,654,2,-1,0,3,1 41,9,2,2,no,145,0,0,0,22,5,1833,2,-1,0,3,1 41,4,0,3,no,465,1,0,0,22,5,1508,1,-1,0,3,1 28,1,1,1,no,751,1,0,0,22,5,598,5,-1,0,3,1 26,7,1,1,no,-379,1,1,0,22,5,1237,7,-1,0,3,1 39,1,2,1,no,0,0,0,0,22,5,864,4,-1,0,3,1 25,0,1,3,no,349,1,0,0,22,5,1037,8,-1,0,3,1 37,0,1,1,no,1110,1,0,1,22,5,802,11,-1,0,3,1 25,4,1,1,no,2228,1,0,0,23,5,754,1,-1,0,3,1 46,4,0,2,no,2166,1,0,0,23,5,870,1,-1,0,3,1 25,1,2,1,no,-247,1,0,0,23,5,633,1,-1,0,3,1 28,9,2,2,no,-123,0,1,0,23,5,690,1,-1,0,3,1 33,7,0,1,no,44,0,1,0,23,5,529,1,-1,0,3,1 32,4,2,1,no,131,1,1,0,23,5,1152,2,-1,0,3,1 32,1,1,1,no,476,1,0,0,23,5,956,9,-1,0,3,1 47,9,2,1,no,1101,1,0,0,23,5,957,3,-1,0,3,1 50,2,1,2,no,-168,1,0,0,23,5,1327,2,-1,0,3,1 27,9,1,1,no,-392,1,0,0,23,5,228,2,-1,0,3,1 45,0,1,1,no,524,1,0,0,23,5,808,1,-1,0,3,1 51,0,2,1,no,895,0,0,0,23,5,638,2,-1,0,3,1 36,0,1,1,no,212,1,1,0,23,5,1201,6,-1,0,3,1 51,6,1,1,no,95,1,0,0,23,5,1309,4,-1,0,3,1 33,7,2,2,no,395,1,0,0,23,5,1359,6,-1,0,3,1 59,9,1,1,no,135,1,1,0,23,5,416,4,-1,0,3,1 38,2,1,2,no,0,0,0,0,23,5,532,2,-1,0,3,1 40,7,1,1,no,471,1,0,0,24,5,430,4,-1,0,3,1 51,1,1,0,no,603,1,0,0,24,5,853,2,-1,0,3,1 41,2,2,1,no,663,0,0,0,24,5,1204,1,-1,0,3,1 38,2,1,1,no,593,1,1,0,24,5,1484,24,-1,0,3,1 31,4,1,2,no,1224,1,1,0,24,5,1441,2,-1,0,3,1 28,0,2,1,no,2,1,0,0,24,5,454,1,-1,0,3,1 41,1,1,1,yes,720,0,1,0,24,5,651,1,-1,0,3,1 33,4,0,2,no,990,1,0,0,24,5,1491,2,-1,0,3,1 25,8,2,1,no,1747,1,0,0,24,5,903,1,-1,0,3,1 45,0,0,1,no,59,1,0,0,24,5,873,8,-1,0,3,1 31,1,1,0,no,33,1,0,0,24,5,901,2,-1,0,3,1 28,7,2,1,no,340,1,0,0,24,5,821,2,-1,0,3,1 49,9,0,1,no,305,0,0,0,24,5,367,2,-1,0,3,1 25,7,1,1,no,0,1,0,0,24,5,458,9,-1,0,3,1 28,10,2,2,no,497,1,0,0,24,5,1602,6,-1,0,3,1 54,5,1,1,no,1371,0,0,1,25,5,1492,2,-1,0,3,1 28,9,2,2,no,1628,1,0,0,25,5,1422,2,-1,0,3,1 27,9,2,2,no,372,1,1,0,25,5,728,2,-1,0,3,1 49,1,2,0,no,566,1,0,0,25,5,979,2,-1,0,3,1 53,1,1,0,no,343,1,0,0,25,5,616,5,-1,0,3,1 26,9,2,1,no,941,0,1,0,25,5,784,1,-1,0,3,1 37,7,1,1,no,775,0,1,0,25,5,1000,1,-1,0,3,1 34,1,2,1,no,1759,0,1,0,25,5,695,1,-1,0,3,1 51,5,1,0,no,1495,0,0,1,25,5,1946,4,-1,0,3,1 39,3,1,1,no,1444,0,0,1,25,5,362,8,-1,0,3,1 35,9,2,2,no,470,1,0,0,25,5,743,13,-1,0,3,1 51,10,1,0,no,1162,1,0,0,25,5,2015,1,-1,0,3,1 30,4,2,2,no,2,1,1,0,25,5,1031,8,-1,0,3,1 42,1,2,0,no,1628,1,0,0,25,5,590,8,-1,0,3,1 54,4,1,1,no,1134,1,0,0,25,5,1330,3,-1,0,3,1 51,1,1,0,no,0,1,1,0,25,5,946,3,-1,0,3,1 57,5,0,0,no,63,0,1,0,25,5,1448,17,-1,0,3,1 30,4,1,2,no,1221,1,0,1,25,5,279,4,-1,0,3,1 37,1,2,1,no,5024,0,1,0,25,5,661,4,-1,0,3,1 39,9,1,1,no,10685,1,0,0,25,5,1369,9,-1,0,3,1 35,9,1,2,no,1629,0,0,0,25,5,653,4,-1,0,3,1 48,4,1,1,no,268,1,0,0,28,5,458,7,-1,0,3,1 31,4,1,2,no,0,1,1,0,28,5,664,8,-1,0,3,1 35,6,1,1,no,5613,1,0,0,28,5,699,3,-1,0,3,1 48,3,1,1,no,513,0,1,0,28,5,939,3,-1,0,3,1 53,3,1,0,no,362,0,0,0,28,5,1169,2,-1,0,3,1 27,9,2,1,no,3933,0,0,0,28,5,837,3,-1,0,3,1 55,9,1,1,no,3899,1,0,1,28,5,596,4,-1,0,3,1 40,7,1,1,no,473,1,0,0,28,5,383,3,-1,0,3,1 40,6,2,2,no,1616,0,0,1,28,5,1009,7,-1,0,3,1 38,1,0,1,no,1817,1,0,0,28,5,1096,4,-1,0,3,1 54,5,2,1,no,129,0,1,0,28,5,867,2,-1,0,3,1 54,9,1,1,no,1938,0,1,0,28,5,551,3,-1,0,3,1 50,0,2,1,no,133,0,0,0,28,5,890,4,-1,0,3,1 48,4,0,2,no,2263,0,0,0,29,5,874,2,-1,0,3,1 41,7,1,3,no,1538,0,0,0,29,5,710,2,-1,0,3,1 51,1,0,0,no,5,0,0,0,29,5,600,2,-1,0,3,1 42,9,2,2,no,2625,0,0,0,29,5,2516,2,-1,0,3,1 58,5,0,1,no,1080,1,0,0,29,5,1058,2,-1,0,3,1 47,10,1,1,no,312,0,0,0,29,5,884,1,-1,0,3,1 58,0,1,1,no,145,1,0,1,29,5,616,13,-1,0,3,1 41,1,1,1,no,-190,0,1,0,29,5,808,3,-1,0,3,1 50,0,1,1,no,2678,0,0,1,30,5,1011,6,-1,0,3,1 46,3,1,1,no,271,1,0,0,30,5,1013,29,-1,0,3,1 44,1,1,1,no,0,0,0,0,30,5,684,2,-1,0,3,1 36,7,1,1,no,0,0,0,0,30,5,843,2,-1,0,3,1 58,4,0,1,no,3161,0,0,1,30,5,542,2,-1,0,3,1 44,10,2,0,yes,-4,0,1,0,30,5,788,2,-1,0,3,1 49,2,0,3,yes,-701,1,0,0,30,5,988,2,-1,0,3,1 55,4,1,2,no,-375,0,0,0,30,5,814,2,-1,0,3,1 52,10,1,0,no,959,0,0,1,30,5,694,2,-1,0,3,1 60,5,0,1,no,1091,0,0,0,30,5,441,5,-1,0,3,1 41,7,1,3,yes,-1,0,1,0,30,5,1171,4,-1,0,3,1 36,4,1,2,no,96,0,0,0,30,5,729,4,-1,0,3,1 41,1,0,0,no,0,0,0,0,30,5,509,2,-1,0,3,1 50,2,1,2,no,230,0,0,1,30,5,1089,4,-1,0,3,1 44,3,1,0,no,-156,0,1,0,30,5,1211,2,-1,0,3,1 54,9,1,1,no,114,0,0,0,30,5,458,3,-1,0,3,1 35,4,1,2,no,-468,1,0,0,30,5,540,8,-1,0,3,1 59,5,0,1,no,679,0,0,0,30,5,761,4,-1,0,3,1 42,2,1,3,no,206,1,0,0,30,5,695,2,-1,0,3,1 27,4,2,2,no,1,1,0,0,30,5,764,4,-1,0,3,1 59,3,0,0,no,2518,0,0,0,30,5,675,3,-1,0,3,1 43,0,1,1,no,378,0,0,0,30,5,782,8,-1,0,3,1 39,9,2,1,no,391,0,1,0,30,5,962,4,-1,0,3,1 33,10,1,1,no,-144,0,0,0,30,5,481,3,-1,0,3,1 36,6,2,2,no,1240,0,0,0,31,5,812,3,-1,0,3,1 35,1,0,0,no,1792,1,0,0,31,5,630,6,-1,0,3,1 50,1,1,3,no,2284,1,0,1,31,5,1088,17,-1,0,3,1 55,9,1,1,no,264,1,1,0,31,5,1150,5,-1,0,3,1 51,10,0,2,no,0,0,0,0,31,5,574,2,-1,0,3,1 48,2,1,2,yes,-392,0,0,0,31,5,725,4,-1,0,3,1 55,3,0,0,no,77,0,0,0,31,5,463,6,-1,0,3,1 30,7,1,1,no,48,1,0,0,31,5,662,13,-1,0,3,1 53,1,1,0,no,252,0,0,0,31,5,727,4,-1,0,3,1 40,1,1,1,no,1954,0,0,0,31,5,1107,2,-1,0,3,1 44,9,1,0,no,755,0,1,0,31,5,829,4,-1,0,3,1 49,4,0,2,no,755,0,0,0,31,5,1212,4,-1,0,3,1 40,0,1,1,no,341,0,0,0,31,5,1142,5,-1,0,3,1 58,10,1,0,no,1183,0,0,0,31,5,1721,5,-1,0,3,1 49,7,1,1,no,388,0,0,1,31,5,1032,4,-1,0,3,1 38,9,1,1,no,1919,0,0,0,4,1,846,1,-1,0,3,1 51,3,0,2,no,1613,0,0,0,4,1,870,1,-1,0,3,1 36,4,0,2,no,0,0,0,0,4,1,1165,1,-1,0,3,1 55,5,1,1,no,0,0,0,0,4,1,1110,1,-1,0,3,1 43,9,1,2,no,0,0,0,0,4,1,1488,2,-1,0,3,1 60,4,0,1,no,0,0,0,0,5,1,603,3,-1,0,3,1 57,5,1,1,no,466,0,0,0,5,1,641,2,-1,0,3,1 41,9,1,2,no,515,0,0,0,5,1,837,2,-1,0,3,1 50,9,1,2,no,2313,0,0,0,5,1,553,1,-1,0,3,1 53,1,1,1,no,-8,0,0,0,5,1,794,1,-1,0,3,1 59,5,1,0,no,1634,1,1,0,5,1,836,1,-1,0,3,1 59,3,1,0,no,100,0,0,0,5,1,1134,1,-1,0,3,1 52,11,1,3,no,5361,0,0,0,5,1,607,3,-1,0,3,1 58,4,0,2,no,1257,0,0,0,5,1,1536,1,-1,0,3,1 59,5,1,0,no,3400,1,0,0,5,1,515,3,-1,0,3,1 49,9,2,1,no,1377,0,0,0,5,1,935,1,-1,0,3,1 33,4,0,2,no,2885,0,0,0,5,1,1051,2,-1,0,3,1 55,1,1,0,no,512,0,0,0,5,1,1200,2,-1,0,3,1 42,9,1,2,no,921,0,0,0,5,1,531,3,-1,0,3,1 52,7,1,1,no,1000,0,0,0,5,1,766,2,-1,0,3,1 38,9,1,2,no,202,0,0,0,5,1,1311,2,-1,0,3,1 55,9,1,2,no,169,0,1,0,5,1,618,2,-1,0,3,1 49,9,1,2,no,3726,0,0,0,5,1,875,2,-1,0,3,1 56,9,0,1,no,146,1,0,0,5,1,419,3,-1,0,3,1 46,9,1,1,no,940,0,0,0,5,1,1227,2,-1,0,3,1 57,5,0,2,no,734,0,0,0,6,1,1357,4,-1,0,3,1 61,5,1,1,no,77,0,0,0,6,1,455,4,-1,0,3,1 45,0,1,1,no,925,0,0,0,6,1,406,2,-1,0,3,1 39,4,1,2,no,1836,0,0,0,6,1,902,3,-1,0,3,1 46,0,1,1,no,660,0,0,0,6,1,740,2,-1,0,3,1 43,4,2,2,no,1304,0,0,0,6,1,501,2,-1,0,3,1 34,7,1,1,no,2956,0,0,0,6,1,835,2,-1,0,3,1 35,4,0,2,no,146,0,0,0,6,1,720,2,-1,0,3,1 36,4,0,2,yes,36,0,0,0,6,1,836,4,-1,0,3,1 45,4,0,2,no,1,0,0,0,6,1,490,2,-1,0,3,1 47,9,1,1,no,2480,0,0,0,6,1,763,2,-1,0,3,1 42,9,1,2,no,14282,1,0,0,6,1,649,4,-1,0,3,1 40,4,2,2,no,0,0,0,0,6,1,641,6,-1,0,3,1 40,9,0,1,no,311,0,0,0,6,1,738,2,-1,0,3,1 32,9,1,1,no,560,0,0,0,6,1,1044,4,-1,0,3,1 33,9,2,1,no,3059,0,1,0,6,1,482,2,-1,0,3,1 32,6,1,2,no,1423,0,0,0,6,1,1249,4,-1,0,3,1 46,1,1,0,no,874,0,0,0,6,1,996,1,-1,0,3,1 50,0,1,1,no,706,1,0,0,6,1,1250,4,-1,0,3,1 51,1,1,0,no,7098,0,0,0,6,1,1471,7,-1,0,3,1 30,4,2,2,no,5389,0,0,0,6,1,1456,3,-1,0,3,1 39,9,2,1,no,1104,0,0,0,7,1,395,2,-1,0,3,1 47,0,1,1,no,3070,0,0,0,7,1,663,4,-1,0,3,1 40,9,0,1,no,0,0,0,0,7,1,674,2,-1,0,3,1 50,1,1,0,no,4108,1,0,0,7,1,526,2,-1,0,3,1 60,4,1,2,no,0,0,0,0,7,1,401,3,-1,0,3,1 32,4,2,2,no,4291,0,0,0,7,1,1321,9,-1,0,3,1 57,9,1,2,no,6822,0,0,0,7,1,797,2,-1,0,3,1 41,6,1,2,no,231,0,0,0,7,1,352,2,-1,0,3,1 50,9,1,1,no,3176,0,0,0,7,1,670,1,-1,0,3,1 46,9,0,1,no,-824,1,0,0,7,1,429,3,-1,0,3,1 31,4,1,2,no,676,0,0,0,7,1,1182,1,-1,0,3,1 36,9,2,1,no,651,0,0,0,7,1,876,1,-1,0,3,1 45,9,0,1,no,2657,0,0,0,7,1,895,3,-1,0,3,1 33,9,0,1,no,801,0,0,0,7,1,341,5,-1,0,3,1 47,7,1,1,no,405,0,1,0,8,1,994,2,-1,0,3,1 35,4,1,2,no,82,0,0,0,8,1,507,4,-1,0,3,1 38,4,1,2,no,179,0,0,2,8,1,317,4,-1,0,3,1 33,4,1,2,no,1047,0,0,0,8,1,512,4,-1,0,3,1 44,9,2,1,no,7138,0,0,0,8,1,809,2,-1,0,3,1 31,4,2,2,no,165,0,1,0,8,1,523,2,-1,0,3,1 31,4,2,2,no,37,0,0,0,8,1,669,2,-1,0,3,1 49,7,1,1,no,823,0,0,0,8,1,780,4,-1,0,3,1 52,7,1,0,no,1625,0,0,0,8,1,459,2,-1,0,3,1 31,9,1,2,no,120,0,0,0,8,1,1000,1,-1,0,3,1 35,4,1,2,no,2957,1,1,0,8,1,733,3,-1,0,3,1 31,4,2,2,no,2,0,0,0,8,1,848,1,-1,0,3,1 36,9,0,1,no,1,0,0,0,8,1,1242,3,-1,0,3,1 39,4,1,2,no,1795,0,0,0,8,1,922,7,-1,0,3,1 46,4,1,2,no,4,0,0,0,8,1,1206,4,-1,0,3,1 48,1,1,0,no,476,0,0,0,11,1,856,3,-1,0,3,1 36,4,1,2,no,9,0,0,0,11,1,386,4,-1,0,3,1 30,4,1,2,no,131,1,0,0,11,1,384,6,-1,0,3,1 33,4,2,2,no,1120,0,0,0,11,1,1070,2,-1,0,3,1 57,3,1,0,no,2785,0,0,0,11,1,610,4,-1,0,3,1 33,4,2,2,no,0,0,0,0,11,1,699,7,-1,0,3,1 35,4,1,2,no,43,0,0,0,11,1,917,4,-1,0,3,1 53,4,1,2,no,0,0,0,0,11,1,608,2,-1,0,3,1 41,4,1,1,no,456,1,0,0,11,1,734,4,-1,0,3,1 32,4,2,1,no,2018,0,0,0,11,1,1238,1,-1,0,3,1 41,4,1,2,no,0,0,0,0,11,1,1242,5,-1,0,3,1 35,9,2,1,no,354,0,0,0,11,1,444,3,-1,0,3,1 48,1,1,0,no,719,0,1,0,11,1,418,5,-1,0,3,1 34,4,1,2,no,828,0,1,0,11,1,1080,3,-1,0,3,1 47,1,1,1,no,2787,0,0,0,11,1,479,1,-1,0,3,1 46,10,1,1,no,2551,0,0,0,12,1,645,2,-1,0,3,1 60,5,1,1,no,2722,0,0,0,12,1,506,2,-1,0,3,1 39,4,1,1,no,1880,0,0,0,12,1,768,2,-1,0,3,1 57,1,1,0,no,2112,0,0,0,12,1,1134,2,-1,0,3,1 36,10,0,1,no,871,0,0,0,12,1,396,3,-1,0,3,1 30,4,1,1,no,850,0,0,0,12,1,507,2,-1,0,3,1 33,9,1,1,no,3243,0,0,0,12,1,439,4,-1,0,3,1 30,4,1,2,no,1567,1,0,0,12,1,1133,4,-1,0,3,1 36,9,2,1,yes,12,0,0,0,12,1,587,2,-1,0,3,1 49,1,1,0,no,128,0,0,0,12,1,696,2,-1,0,3,1 49,1,1,1,no,553,0,0,0,12,1,645,8,-1,0,3,1 40,10,0,1,no,1694,0,0,0,12,1,473,6,-1,0,3,1 49,1,1,0,no,408,0,0,0,12,1,559,4,-1,0,3,1 38,4,1,2,no,1223,0,0,0,12,1,1092,5,-1,0,3,1 31,4,2,2,no,1858,0,0,0,12,1,453,3,-1,0,3,1 34,4,1,2,no,520,1,0,0,12,1,1307,4,-1,0,3,1 39,4,1,2,no,271,1,0,0,12,1,1344,1,-1,0,3,1 40,4,1,1,no,4396,0,0,0,12,1,432,3,-1,0,3,1 47,9,1,1,no,568,1,0,0,12,1,1613,1,-1,0,3,1 37,9,2,1,no,3665,0,0,0,12,1,664,3,-1,0,3,1 44,9,2,1,no,244,1,0,0,12,1,1735,4,-1,0,3,1 55,4,1,2,no,568,0,0,0,12,1,940,10,-1,0,3,1 40,9,1,1,no,42,0,0,0,13,1,1842,2,-1,0,3,1 54,5,1,0,no,309,0,0,0,13,1,969,2,-1,0,3,1 35,9,1,1,no,-180,0,1,0,13,1,823,2,-1,0,3,1 32,9,2,2,no,34646,0,0,1,13,1,618,9,-1,0,3,1 36,4,0,2,no,59,0,0,0,13,1,526,2,-1,0,3,1 37,4,1,2,no,4436,0,0,0,13,1,846,5,-1,0,3,1 37,9,2,1,no,-242,1,0,0,13,1,1149,6,-1,0,3,1 50,6,1,1,no,10052,0,0,0,13,1,665,2,-1,0,3,1 58,5,1,2,no,0,0,0,0,13,1,641,4,-1,0,3,1 52,9,0,1,no,152,0,0,0,13,1,404,6,-1,0,3,1 31,4,1,2,no,156,0,0,0,13,1,657,7,-1,0,3,1 40,4,1,2,no,66,0,0,0,13,1,737,4,-1,0,3,1 34,4,2,2,no,9827,1,0,0,13,1,871,4,-1,0,3,1 53,7,1,1,no,6170,0,0,0,13,1,838,4,-1,0,3,1 43,4,1,2,no,2383,0,0,0,13,1,379,6,-1,0,3,1 30,9,2,2,no,635,0,0,0,13,1,851,8,-1,0,3,1 34,9,2,1,no,16,0,0,0,13,1,990,1,-1,0,3,1 48,0,0,1,no,362,0,0,0,13,1,529,1,-1,0,3,1 33,4,1,2,no,191,0,0,0,13,1,1148,2,-1,0,3,1 37,4,2,2,no,455,1,0,0,13,1,904,6,-1,0,3,1 57,9,0,1,no,3043,0,0,0,14,1,707,2,-1,0,3,1 55,5,1,0,no,3334,0,0,0,14,1,632,4,-1,0,3,1 34,9,1,2,no,8029,1,0,0,14,1,593,4,-1,0,3,1 44,2,1,2,no,624,0,0,1,14,1,552,2,-1,0,3,1 35,4,2,2,no,331,0,0,0,14,1,857,2,-1,0,3,1 34,3,1,0,no,2929,0,0,0,14,1,518,2,-1,0,3,1 49,7,1,1,no,1314,0,0,0,14,1,834,14,-1,0,3,1 30,9,1,1,no,289,1,0,0,14,1,1184,4,-1,0,3,1 32,4,2,2,no,20,0,0,0,14,1,1151,2,-1,0,3,1 56,1,1,0,no,8163,0,0,0,14,1,1231,4,-1,0,3,1 50,4,2,2,no,1200,0,0,0,14,1,904,3,-1,0,3,1 32,9,2,1,no,1732,1,0,0,14,1,1871,3,-1,0,3,1 ``` The content has been capped at 50000 tokens, and files over NaN bytes have been omitted. The user could consider applying other filters to refine the result. The better and more specific the context, the better the LLM can follow instructions. If the context seems verbose, the user can refine the filter using uithub. Thank you for using https://uithub.com - Perfect LLM context for any GitHub repo.