{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Example I: Tracking data stored in files\n", "\n", "In this example notebook we are going to track rain-rates that are estimated from radar based reflectivities. The rain-rates is an example dataset from the CPOL radar archive that used to operate north of Darwin, Australia. This example dataset has been provided by Valentin Louf from the Australian Bureau of Meteorology.\n", "\n", "A more detailed example of the application of the tracking algorithm can be found in [doi: 10.1002/qj.4360](https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/qj.4360)\n", "\n", "Before we get started we import all modules that are needed to apply the tracking" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from IPython.display import HTML\n", "from tintx import RunDirectory, config\n", "import xarray as xr\n", "import pandas as pd\n", "from pathlib import Path\n", "import os\n", "import matplotlib.pyplot as plt\n", "import cartopy.crs as crs\n", "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example we want to apply the tracking algorithm only for a limited time period. Such time periods can be passed as string or `datetime` object. If defined as a string make sure they follow the [ISO 8061](https://en.wikipedia.org/wiki/ISO_8601) convention." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "first = '2006-11-16 03:00' #Start-date\n", "last = '2006-11-16 11:00' #End-date\n", "data_files = Path(os.environ[\"DATA_FILES\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To apply the tracking we create an instance of the `RunDirectory` class. Normally we would have to create this instance with a `xarray.DataArray`. The `from_files` method is a convenience method that allows us to create this instance without opening a dataset. The method will open the data files and create an object. We will have to provide a filename or a list of files, the name of the variable and if not according to cf conventions the names of the *longitude*, *latitude* and *time* dimensions and the used Coordinate Reference System (CRS) of the source data:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "RD = RunDirectory.from_files( data_files / \"CPOL_radar.nc\", \"radar_estimated_rain_rate\",\n", " start=first, end=last, use_cftime=True, x_coord=\"longitude\", y_coord=\"latitude\", crs=\"aeqd\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Additional keyword arguments can be added to the `xarray.open_mfdataset` method. In this example we used the `use_cftime` keyword to convert the time variable to cftime vectors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The tuning parameters\n", "To apply the actual tracking algorithm we can use the `get_tracks` method. The algorithm can be tuned by passing the values for the tuning parameters. See also the [tuning parameter section](api.html#tracking-parameter-guide) of the docs." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Tracking tuning parameters\n", "--------------------------\n", "\n", "The following parameter can be set to tune the cell tracking algorithm\n", "\n", "* field_thresh: units of 'field' attribute, default: 32\n", " The threshold used for object detection. Detected objects are connected\n", " pixels above this threshold.\n", "* iso_thresh: units of 'field' attribute, default: 4\n", " Used in isolated cell classification. Isolated cells must not be connected\n", " to any other cell by contiguous pixels above this threshold.\n", "* iso_smooth: pixels, default: 4\n", " Gaussian smoothing parameter in peak detection preprocessing. See\n", " single_max in tint.objects.\n", "* min_size: square kilometers, default: 8\n", " The minimum size threshold in pixels for an object to be detected.\n", "* search_margin: meters, default: 250\n", " The radius of the search box around the predicted object center.\n", "* flow_margin: meters, default: 750\n", " The margin size around the object extent on which to perform phase\n", " correlation.\n", "* max_disparity: float, default: 999\n", " Maximum allowable disparity value. Larger disparity values are sent to a\n", " large number.\n", "* max_flow_mag: meters per second, default: 50\n", " Maximum allowable global shift magnitude. See get_global_shift in\n", " tint.phase_correlation.\n", "* max_shift_disp: meters per second, default: 15\n", " Maximum magnitude of difference in meters per second for two shifts to be\n", " considered in agreement. See correct_shift in tint.matching.\n", "* gs_alt: meters, default: 1500\n", " Altitude in meters at which to perform phase correlation for global shift\n", " calculation. See correct_shift in tint.matching.\n", "\n", "\n", "Setting and getting the tuning parameters\n", "-----------------------------------------\n", "The parameters can the set using the :py:class:`tintx.config.set` class.\n", "Getting the values of the currently set parameters can be done by the\n", ":py:func:`tintx.config.get` method.\n", "\n" ] } ], "source": [ "print(config.__doc__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Apply the tracking with a rain-rate threshold of 0.1 and a minimum size of 4. The return value of the function will be the total number of individual storms identified by the algorithm." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "a2b02b6c1fb348cdacdfe4352bc72dcd", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Tracking: 0%| | 0/48 [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Number of storm cells found: 125\n" ] } ], "source": [ "num_cells = RD.get_tracks(field_thresh=0.1, min_size=4)\n", "print(f\"Number of storm cells found: {num_cells}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Accessing the track data\n", "The actual tracking data is stored in a multi-index [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.MultiIndex.html) and can be accessed via the `tracks` property, it will be converted to [geopandas.GeoDataFrame](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.html) on the fly:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
| \n", " | \n", " | time | \n", "grid_x | \n", "grid_y | \n", "lon | \n", "lat | \n", "area | \n", "max | \n", "mean | \n", "isolated | \n", "geometry | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| scan | \n", "uid | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 0 | \n", "0 | \n", "2006-11-16 03:00:00 | \n", "6.308 | \n", "45.731 | \n", "129.8469 | \n", "-12.5164 | \n", "26 | \n", "1.451491 | \n", "0.980084 | \n", "False | \n", "POLYGON ((-130000.000 -40000.000, -125000.000 ... | \n", "
| 1 | \n", "2006-11-16 03:00:00 | \n", "2.034 | \n", "51.655 | \n", "129.7554 | \n", "-12.3810 | \n", "29 | \n", "1.398994 | \n", "0.771455 | \n", "False | \n", "POLYGON ((-140000.000 -27500.000, -135000.000 ... | \n", "|
| 2 | \n", "2006-11-16 03:00:00 | \n", "103.341 | \n", "50.902 | \n", "132.0804 | \n", "-12.4046 | \n", "41 | \n", "5.279258 | \n", "0.977871 | \n", "False | \n", "POLYGON ((107500.000 -25000.000, 110000.000 -2... | \n", "|
| 3 | \n", "2006-11-16 03:00:00 | \n", "75.000 | \n", "49.750 | \n", "131.4358 | \n", "-12.4288 | \n", "4 | \n", "1.249542 | \n", "0.653577 | \n", "False | \n", "POLYGON ((42500.000 -22500.000, 45000.000 -225... | \n", "|
| 4 | \n", "2006-11-16 03:00:00 | \n", "90.684 | \n", "52.842 | \n", "131.8040 | \n", "-12.3605 | \n", "19 | \n", "3.560659 | \n", "1.146169 | \n", "False | \n", "POLYGON ((82500.000 -17500.000, 87500.000 -175... | \n", "|
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 46 | \n", "121 | \n", "2006-11-16 10:40:00 | \n", "37.000 | \n", "78.630 | \n", "130.5622 | \n", "-11.7766 | \n", "27 | \n", "31.208767 | \n", "7.085050 | \n", "True | \n", "POLYGON ((-62500.000 47500.000, -62500.000 525... | \n", "
| 124 | \n", "2006-11-16 10:40:00 | \n", "30.800 | \n", "79.600 | \n", "130.4244 | \n", "-11.7539 | \n", "5 | \n", "9.346570 | \n", "2.719892 | \n", "True | \n", "POLYGON ((-70000.000 52500.000, -70000.000 575... | \n", "|
| 47 | \n", "123 | \n", "2006-11-16 10:50:00 | \n", "89.444 | \n", "15.333 | \n", "131.7603 | \n", "-13.2150 | \n", "9 | \n", "0.227352 | \n", "0.168942 | \n", "False | \n", "POLYGON ((77500.000 -110000.000, 82500.000 -11... | \n", "
| 121 | \n", "2006-11-16 10:50:00 | \n", "37.400 | \n", "78.640 | \n", "130.5622 | \n", "-11.7766 | \n", "25 | \n", "14.703703 | \n", "3.166035 | \n", "False | \n", "POLYGON ((-60000.000 47500.000, -60000.000 525... | \n", "|
| 48 | \n", "121 | \n", "2006-11-16 11:00:00 | \n", "36.905 | \n", "79.095 | \n", "130.5622 | \n", "-11.7766 | \n", "21 | \n", "14.791470 | \n", "3.332414 | \n", "True | \n", "POLYGON ((-62500.000 47500.000, -62500.000 500... | \n", "
299 rows × 10 columns
\n", "<xarray.Dataset>\n",
"Dimensions: (time: 49, x: 117, y: 117)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2006-11-16T03:00:00 ... ...\n",
" * x (x) int32 -145000 -142500 ... 142500 145000\n",
" * y (y) int32 -145000 -142500 ... 142500 145000\n",
"Data variables:\n",
" radar_estimated_rain_rate (time, x, y) float64 dask.array<chunksize=(49, 117, 117), meta=np.ndarray>\n",
" latitude (x, y) float64 dask.array<chunksize=(117, 117), meta=np.ndarray>\n",
" longitude (x, y) float64 dask.array<chunksize=(117, 117), meta=np.ndarray>\n",
" isfile (time) int32 dask.array<chunksize=(49,), meta=np.ndarray>\n",
"Attributes: (12/25)\n",
" Conventions: CF/Radial instrument_parameters\n",
" version: 2017-10\n",
" title: CPOL product level 2\n",
" institution: Australia Bureau of Meteorology\n",
" references: If you use this dataset, please cite: 'An integrated...\n",
" source: UF raw files\n",
" ... ...\n",
" state: NT\n",
" history: October 2017 recalibration: May 2017 recalibration\n",
" volume_number: 0\n",
" platform_type: fixed\n",
" instrument_type: radar\n",
" primary_axis: axis_z| \n", " | \n", " | time | \n", "grid_x | \n", "grid_y | \n", "lon | \n", "lat | \n", "area | \n", "max | \n", "mean | \n", "isolated | \n", "geometry | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|
| scan | \n", "uid | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
| 0 | \n", "0 | \n", "2006-11-16 03:00:00 | \n", "6.308 | \n", "45.731 | \n", "129.8469 | \n", "-12.5164 | \n", "26 | \n", "1.451491 | \n", "0.980084 | \n", "False | \n", "POLYGON ((-130000.000 -40000.000, -125000.000 ... | \n", "
| 1 | \n", "2006-11-16 03:00:00 | \n", "2.034 | \n", "51.655 | \n", "129.7554 | \n", "-12.3810 | \n", "29 | \n", "1.398994 | \n", "0.771455 | \n", "False | \n", "POLYGON ((-140000.000 -27500.000, -135000.000 ... | \n", "|
| 2 | \n", "2006-11-16 03:00:00 | \n", "103.341 | \n", "50.902 | \n", "132.0804 | \n", "-12.4046 | \n", "41 | \n", "5.279258 | \n", "0.977871 | \n", "False | \n", "POLYGON ((107500.000 -25000.000, 110000.000 -2... | \n", "|
| 3 | \n", "2006-11-16 03:00:00 | \n", "75.000 | \n", "49.750 | \n", "131.4358 | \n", "-12.4288 | \n", "4 | \n", "1.249542 | \n", "0.653577 | \n", "False | \n", "POLYGON ((42500.000 -22500.000, 45000.000 -225... | \n", "|
| 4 | \n", "2006-11-16 03:00:00 | \n", "90.684 | \n", "52.842 | \n", "131.8040 | \n", "-12.3605 | \n", "19 | \n", "3.560659 | \n", "1.146169 | \n", "False | \n", "POLYGON ((82500.000 -17500.000, 87500.000 -175... | \n", "|
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 46 | \n", "121 | \n", "2006-11-16 10:40:00 | \n", "37.000 | \n", "78.630 | \n", "130.5622 | \n", "-11.7766 | \n", "27 | \n", "31.208767 | \n", "7.085050 | \n", "True | \n", "POLYGON ((-62500.000 47500.000, -62500.000 525... | \n", "
| 124 | \n", "2006-11-16 10:40:00 | \n", "30.800 | \n", "79.600 | \n", "130.4244 | \n", "-11.7539 | \n", "5 | \n", "9.346570 | \n", "2.719892 | \n", "True | \n", "POLYGON ((-70000.000 52500.000, -70000.000 575... | \n", "|
| 47 | \n", "123 | \n", "2006-11-16 10:50:00 | \n", "89.444 | \n", "15.333 | \n", "131.7603 | \n", "-13.2150 | \n", "9 | \n", "0.227352 | \n", "0.168942 | \n", "False | \n", "POLYGON ((77500.000 -110000.000, 82500.000 -11... | \n", "
| 121 | \n", "2006-11-16 10:50:00 | \n", "37.400 | \n", "78.640 | \n", "130.5622 | \n", "-11.7766 | \n", "25 | \n", "14.703703 | \n", "3.166035 | \n", "False | \n", "POLYGON ((-60000.000 47500.000, -60000.000 525... | \n", "|
| 48 | \n", "121 | \n", "2006-11-16 11:00:00 | \n", "36.905 | \n", "79.095 | \n", "130.5622 | \n", "-11.7766 | \n", "21 | \n", "14.791470 | \n", "3.332414 | \n", "True | \n", "POLYGON ((-62500.000 47500.000, -62500.000 500... | \n", "
299 rows × 10 columns
\n", "