Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
db21278
add first quadtree construction
mathleur Jan 2, 2024
f9ca22c
remove unnecessary code
mathleur Jan 2, 2024
7fd6cb2
add first quad tree construction in slicer
mathleur Jan 2, 2024
a6b7fe1
start query polygon function
mathleur Jan 3, 2024
96a2467
add polygon slicing
mathleur Jan 4, 2024
19ca294
write all methods for quadtree slicer
mathleur Jan 4, 2024
e2fa53b
quadtree slicer tests
mathleur Jan 5, 2024
33e6e97
clean up
mathleur Jan 5, 2024
5ffd68a
refactor
mathleur Jan 8, 2024
a941be2
finish quad tree extract function
mathleur Jan 8, 2024
badae88
first performance test for quadtree
mathleur Jan 8, 2024
2dda6df
clean up
mathleur Jan 8, 2024
85b9b21
fix MAX number of nodes in QuadTree
mathleur Jan 9, 2024
a69cad0
optimise quadtree slicer to not slice quadrants when the polygon is a…
mathleur Jan 10, 2024
47024c3
clean up
mathleur Jan 10, 2024
b152235
Merge branch 'develop' of github.com:ecmwf/polytope into feature/quad…
mathleur Jan 10, 2024
b860b0d
pull develop and fix tests
mathleur Jan 10, 2024
c2e9df5
add index in quad node
mathleur Jan 10, 2024
97cabac
add index in quadtree construction from point cloud
mathleur Jan 10, 2024
5fedb73
extract in quadtree slicer using point indices instead of lat/lon
mathleur Jan 10, 2024
811e5f9
reformat
mathleur Jan 10, 2024
b571028
start creating a higher-level extract function to allow for different…
mathleur Jan 15, 2024
7ce4957
add higher level extract inside the Polytope class
mathleur Jan 15, 2024
dcc7399
make new polytope class extract work with old tests
mathleur Jan 16, 2024
be4ebdb
add find_point_cloud to xarray datacube
mathleur Jan 16, 2024
d33fa74
quadtree slicer create lat/lon in index tree and stashes the point in…
mathleur Jan 17, 2024
e967f1a
merge mappings on separate files branch
mathleur Jan 17, 2024
c356e70
add irregular grid mapper support for xarray backend
mathleur Jan 17, 2024
a959b71
make first test with irregular grid for xarray backend work
mathleur Jan 19, 2024
d0dd2e0
make irregular grid work with fdb backend
mathleur Jan 22, 2024
8ba1cbf
fix polytope extract test with xarray backend and irregular mapper
mathleur Jan 23, 2024
a261ea5
fix polytope extract test with xarray backend and irregular mapper
mathleur Jan 23, 2024
8b76621
pull develop
mathleur Jan 25, 2024
be3e74b
fix edge case
mathleur Feb 22, 2024
e4334c9
fix edge cases tests
mathleur Feb 22, 2024
0acb8dc
merge with develop
mathleur Feb 23, 2024
10886c7
clean up
mathleur Feb 23, 2024
b0c20bd
fix flake8
mathleur Feb 23, 2024
4d49503
mark tests using gribjump
mathleur Feb 23, 2024
ccd4498
fix missing gribjump tests
mathleur Feb 23, 2024
04a8a74
fix missing gribjump tests
mathleur Feb 23, 2024
9fb93c9
remove unnecessary file
mathleur Mar 15, 2024
d201087
add ORCA grid example
mathleur Mar 20, 2024
bc9596a
merge develop
mathleur Mar 20, 2024
74702c7
Merge pull request #120 from ecmwf/feature/irregular_grid_merged_develop
mathleur Mar 20, 2024
12b0eba
black and isort
mathleur Mar 20, 2024
85e1741
timings for irregular grid
mathleur Mar 21, 2024
29a4c7e
update gitignore
mathleur Jul 15, 2024
3043dc2
merge irregular grid branch WIP
mathleur Jul 16, 2024
ff33e31
fix non-irregular grid tests
mathleur Jul 16, 2024
cab5ed1
WIP make irregular grid work
mathleur Jul 17, 2024
2a72dc5
fix irregular grid
mathleur Jul 17, 2024
d36e2a2
small optimisation
mathleur Jul 17, 2024
cdc13fa
optimise quad tree construction
mathleur Jul 17, 2024
6046cd9
optimise quad tree
mathleur Jul 17, 2024
f7981d1
fix tests back
mathleur Jul 17, 2024
9425e73
optimise quad tree
mathleur Jul 17, 2024
0558ac8
optimise
mathleur Jul 17, 2024
0f337f6
clean up quad tree
mathleur Jul 17, 2024
c590e85
optimise quad tree
mathleur Jul 18, 2024
a2028e7
optimise and clean up
mathleur Jul 18, 2024
809d788
clean up
mathleur Jul 18, 2024
98b618d
clean up
mathleur Jul 18, 2024
4b99770
optimise more
mathleur Jul 18, 2024
fa70c55
clena up
mathleur Jul 18, 2024
142fd3f
rename module
mathleur Dec 19, 2024
ac86716
merge develop and fix errors
mathleur Dec 31, 2024
0717920
finish fixing all errors
mathleur Jan 2, 2025
db6dd32
clean up
mathleur Jan 2, 2025
e5a5de7
clean up
mathleur Jan 2, 2025
840cc68
Merge branch 'develop' of github.com:ecmwf/polytope into feature/irre…
mathleur Jan 2, 2025
6f8a0d7
fix flake8
mathleur Jan 2, 2025
a984b64
Merge pull request #297 from ecmwf/feature/irregular_grid_w_new_develop
mathleur Jan 2, 2025
169af74
add other unstructured grid example
mathleur Jan 10, 2025
af6fb90
Merge branch 'feature/try_to_merge_irregular_grid_develop' of github.…
mathleur Jan 10, 2025
4225e94
try to add icon grid example
mathleur Jan 10, 2025
1ca57cc
fix bug if polygon is in union, we can remove the right compressed axis
mathleur Jan 13, 2025
6852051
try to add unstructured grid on fdb backend
mathleur Jan 16, 2025
4a06281
make unstructured grid work with FDB and add example
mathleur Jan 16, 2025
d81dd66
add rust quadtree
mathleur Feb 7, 2025
00ba795
compile rust and update gitignore
mathleur Feb 7, 2025
9f1e505
add python wrap to rust code
mathleur Feb 7, 2025
3f7d1b2
modify gitignore
mathleur Feb 7, 2025
4038a40
expose quadpoint and find_nodes_in function to python from rust
mathleur Feb 10, 2025
2af2842
add rust python interface
mathleur Feb 11, 2025
d4f6d3a
use quadtree in rust to extract data
mathleur Feb 11, 2025
8918bdb
try to take references instead of copying rust children
mathleur Feb 12, 2025
23cf10b
try to take references instead of copying rust children
mathleur Feb 12, 2025
244d15d
clean up quadtree rust
mathleur Feb 14, 2025
6081dd7
NOT WORKING - try to only give back point indexes to python from rust
mathleur Feb 14, 2025
4feecae
fix bug - STILL EXTRACTION IS SLOW
mathleur Feb 14, 2025
d0d5eea
start rewriting rust quadtree as an arena
mathleur Feb 17, 2025
fb95423
finish rewriting quadtree as an arena
mathleur Feb 18, 2025
4509c54
add quadtree arena with python integration into polytope
mathleur Feb 19, 2025
183aea1
change rust arena to not use arc mutexes
mathleur Feb 19, 2025
eebe402
only expose necessary functions to python
mathleur Feb 19, 2025
f2c774d
improve time to build quadtree
mathleur Feb 19, 2025
827d386
faster rust quadtree arena
mathleur Feb 19, 2025
25bcd60
make quadtree slightly faster to extract
mathleur Feb 19, 2025
4a717ce
fix tests
mathleur Feb 19, 2025
2da14ac
clean up
mathleur Feb 20, 2025
bf186d6
try not copying rust
mathleur Mar 1, 2025
5806a1d
add rust profiler
mathleur Mar 3, 2025
bbeedce
try to make quadtree optimisation work again
mathleur Mar 3, 2025
fb72742
fix optimisation step
mathleur Mar 3, 2025
d0dbf64
optimise code
mathleur Mar 4, 2025
277c1af
change test
mathleur Mar 4, 2025
320ac7d
add performance measures + plots
mathleur Mar 4, 2025
64ddb6e
plot only the polytope time without gj
mathleur Mar 5, 2025
32e9e62
add more
mathleur Mar 5, 2025
4160d85
try to not store the point in separate Quadpoints
mathleur Mar 6, 2025
e9e78fa
start translating the slicing into rust too
mathleur Mar 6, 2025
a90817f
add qhull test + translate more
mathleur Mar 7, 2025
e2fb738
add QHull function
mathleur Mar 7, 2025
d20afc3
add query_polygon function and error handling to python
mathleur Mar 10, 2025
4b0840b
make quadtree slicer work in rust
mathleur Mar 11, 2025
0e65b59
make rust code query polygon faster
mathleur Mar 11, 2025
9329e14
use custom qhull library
mathleur Mar 12, 2025
0d0ba80
improve complete rust performance
mathleur Mar 12, 2025
71beff8
optimise rust quadtree query polygon
mathleur Mar 12, 2025
17097f0
add TODO comment
mathleur Mar 14, 2025
d43c691
WIP
mathleur Mar 14, 2025
f89b156
try to speed up quadtree query polygon
mathleur Mar 17, 2025
a5a1bc8
WIP not working make quadtree slicing faster
mathleur Apr 1, 2025
5e1e44b
make code work and faster
mathleur Apr 4, 2025
e8d5f36
WIP: try to make faster quadtree query polygon by changing to rust ar…
mathleur Apr 4, 2025
63d8d3e
make rust poylgon slicing faster
mathleur Apr 7, 2025
03fc194
make the combined grid node work with gj extraction
mathleur Apr 8, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
13 changes: 11 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ site
*.grib
*.gif
*.html
*.req
.mypy_cache
example_eo
example_mri
.mypy_cache
*.req
polytope_python.egg-info
new_test_venv_polytope
new_venv_polytope
Expand All @@ -26,3 +26,12 @@ new_updated_numpy_venv
newest-polytope-venv
serializedTree
new_polytope_venv

newest-polytope-venv
*.pstats
*.profile
new_polytope_venv
*.json
polytope_feature/datacube/quadtree/target
venv_python3_11
*.txt
6 changes: 3 additions & 3 deletions examples/3D_shipping_route.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
import numpy as np
from earthkit import data

from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Ellipsoid, Path, Select
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Ellipsoid, Path, Select


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/3D_shipping_route_wave_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
import pytest
from eccodes import codes_grib_find_nearest, codes_grib_new_from_file

from polytope.datacube.backends.fdb import FDBDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Ellipsoid, Path, Select
from polytope_feature.datacube.backends.fdb import FDBDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Ellipsoid, Path, Select
from tests.helper_functions import download_test_data


Expand Down
8 changes: 4 additions & 4 deletions examples/4D_flight_path.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
from earthkit import data
from PIL import Image

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Path, Select
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Path, Select


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/country_slicing.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
from earthkit import data
from shapely.geometry import shape

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Polygon, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Polygon, Select, Union


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/cyclic_route_around_earth.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
import numpy as np
from earthkit import data

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, PathSegment, Select
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, PathSegment, Select


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/healpix_grid_box_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
from earthkit import data
from eccodes import codes_grib_find_nearest, codes_grib_new_from_file

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select


class TestOctahedralGrid:
Expand Down
8 changes: 4 additions & 4 deletions examples/octahedral_grid_box_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
from eccodes import codes_grib_find_nearest, codes_grib_new_from_file
from matplotlib import markers

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select


def find_nearest_latlon(grib_file, target_lat, target_lon):
Expand Down
8 changes: 4 additions & 4 deletions examples/octahedral_grid_country_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
from matplotlib import markers
from shapely.geometry import shape

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Polygon, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Polygon, Select, Union


def find_nearest_latlon(grib_file, target_lat, target_lon):
Expand Down
4 changes: 2 additions & 2 deletions examples/read_me_example.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import numpy as np
from earthkit import data

from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select

ds = data.from_source("file", "./examples/data/winds.grib")
array = ds.to_xarray()
Expand Down
8 changes: 4 additions & 4 deletions examples/slicing_all_ecmwf_countries.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
from earthkit import data
from shapely.geometry import shape

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Polygon, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Polygon, Select, Union


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/timeseries_example.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
from earthkit import data
from shapely.geometry import shape

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Polygon, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Polygon, Select, Union


class Test:
Expand Down
8 changes: 4 additions & 4 deletions examples/wind_farms.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@
from osgeo import gdal
from shapely.geometry import shape

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Polygon, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Polygon, Select, Union


class Test:
Expand Down
8 changes: 4 additions & 4 deletions performance/fdb_performance.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

import pandas as pd

from polytope.datacube.backends.fdb import FDBDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select
from polytope_feature.datacube.backends.fdb import FDBDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select


class TestSlicingFDBDatacube:
Expand Down
8 changes: 4 additions & 4 deletions performance/fdb_performance_3D.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

import pandas as pd

from polytope.datacube.backends.fdb import FDBDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select, Span
from polytope_feature.datacube.backends.fdb import FDBDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select, Span


class TestSlicingFDBDatacube:
Expand Down
6 changes: 2 additions & 4 deletions performance/fdb_slice_many_numbers_timeseries.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,12 @@
import pandas as pd
import pygribjump as gj

from polytope.polytope import Polytope, Request
from polytope.shapes import All, Point, Select
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import All, Point, Select

time1 = time.time()
# Create a dataarray with 3 labelled axes using different index types

# config = {"class": "od", "expver": "0001", "levtype": "sfc", "type": "pf"}
options = {
"axis_config": [
{"axis_name": "step", "transformations": [{"name": "type_change", "type": "int"}]},
Expand Down Expand Up @@ -62,7 +61,6 @@
Select("class", ["od"]),
Select("stream", ["enfo"]),
Select("type", ["pf"]),
# Select("latitude", [0.035149384216], method="surrounding"),
Point(["latitude", "longitude"], [[0.04, 0]], method="surrounding"),
All("number"),
)
Expand Down
6 changes: 3 additions & 3 deletions performance/performance_many_num_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
import pandas as pd
from earthkit import data

from polytope.datacube.backends.fdb import FDBDatacube
from polytope.polytope import Polytope, Request
from polytope.shapes import All, Point, Select, Span
from polytope_feature.datacube.backends.fdb import FDBDatacube
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import All, Point, Select, Span

time1 = time.time()
# Create a dataarray with 3 labelled axes using different index types
Expand Down
8 changes: 4 additions & 4 deletions performance/scalability_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
import numpy as np
import xarray as xr

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Disk, Ellipsoid, Select
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Disk, Ellipsoid, Select


class Test:
Expand Down
8 changes: 4 additions & 4 deletions performance/scalability_test_2.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
import numpy as np
import xarray as xr

from polytope.datacube.backends.xarray import XArrayDatacube
from polytope.engine.hullslicer import HullSlicer
from polytope.polytope import Polytope, Request
from polytope.shapes import Box, Select, Union
from polytope_feature.datacube.backends.xarray import XArrayDatacube
from polytope_feature.engine.hullslicer import HullSlicer
from polytope_feature.polytope import Polytope, Request
from polytope_feature.shapes import Box, Select, Union


class Test:
Expand Down
Loading