Repository navigation
[GH-2095] Geopandas: Implement Scalable workaround and store as geometries instead of EWKB - #2121
Conversation
…ad of ewkb, and fixing up the tests
cc6484f to
7553142
Compare
aaf3fb8 to
cc5d946
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR implements significant scalability improvements and changes to geopandas support by storing geometries natively instead of as EWKB format, requiring Shapely 2.0+, and fixing test cases to handle edge cases that are no longer supported.
- Reimplemented GeoSeries construction to avoid calling
to_pandas()for better scalability - Changed geometry storage from EWKB back to native geometry format to properly access SRID information
- Added Shapely 2.0+ requirement and corresponding test skips for older versions
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| python/tests/geopandas/test_sjoin.py | Added Shapely 2.0+ version requirement check |
| python/tests/geopandas/test_match_geopandas_series.py | Added version check and updated test data to avoid edge cases |
| python/tests/geopandas/test_match_geopandas_dataframe.py | Added Shapely 2.0+ version requirement check |
| python/tests/geopandas/test_geoseries.py | Added version check and commented out LinearRing test cases |
| python/tests/geopandas/test_geodataframe.py | Added version check and reorganized constructor tests |
| python/sedona/geopandas/geoseries.py | Major refactoring to handle geometry types natively and implement scalable construction |
| python/sedona/geopandas/geodataframe.py | Simplified area property and removed EWKB conversion logic |
Comments suppressed due to low confidence (3)
python/tests/geopandas/test_match_geopandas_series.py:330
- The test data was modified to avoid edge cases with arrays starting with None, but there's no accompanying test to verify that this edge case is properly handled or documented as unsupported.
data = [Point(1, 1), None, None, None, Point(0, 1)]
python/tests/geopandas/test_geoseries.py:233
- Similar to the other test file, the test data was simplified to avoid edge cases, but there's no test coverage to verify the behavior of the removed edge case.
data = [Point(0, 0), None]
python/sedona/geopandas/geoseries.py:186
- [nitpick] The error message could be more helpful by suggesting what types are expected or how to convert the data to a compatible format.
raise TypeError(
| LineString([(0, 0), (1, 1), (0, 0)]), | ||
| LinearRing([(0, 0), (1, 1), (1, 0), (0, 1), (0, 0)]), | ||
| LinearRing([(0, 0), (-1, 1), (-1, -1), (1, -1)]), | ||
| # Errors for LinearRing: issue #2120 |
There was a problem hiding this comment.
The comment references issue #2120 but doesn't explain why LinearRing causes errors or what the expected behavior should be. This makes it difficult for future maintainers to understand the limitation.
| # Errors for LinearRing: issue #2120 | |
| # LinearRing geometries are excluded due to known issues with | |
| # compatibility in GeoSeries.is_simple. Specifically, LinearRing | |
| # objects can cause unexpected errors when processed. See issue #2120 | |
| # for more details. The expected behavior is that LinearRing objects | |
| # should be handled similarly to LineString objects without errors. |
| first_idx = ps_series.first_valid_index() | ||
| if first_idx is not None: | ||
| geom = ps_series.iloc[int(first_idx)] | ||
| srid = shapely.get_srid(geom) | ||
|
|
||
| # Shapely objects stored in the ps.Series retain their srid | ||
| # but the GeoSeries does not, so we manually re-set it here | ||
| if srid > 0: | ||
| result.set_crs(srid, inplace=True) |
There was a problem hiding this comment.
The code accesses the first valid geometry to extract SRID information, but doesn't handle the case where geometries in the same series might have different SRIDs, which could lead to inconsistent behavior.
| first_idx = ps_series.first_valid_index() | |
| if first_idx is not None: | |
| geom = ps_series.iloc[int(first_idx)] | |
| srid = shapely.get_srid(geom) | |
| # Shapely objects stored in the ps.Series retain their srid | |
| # but the GeoSeries does not, so we manually re-set it here | |
| if srid > 0: | |
| result.set_crs(srid, inplace=True) | |
| valid_geometries = ps_series.dropna() | |
| srids = {shapely.get_srid(geom) for geom in valid_geometries if shapely.get_srid(geom) > 0} | |
| if len(srids) == 1: | |
| # All SRIDs are consistent, set the CRS | |
| result.set_crs(srids.pop(), inplace=True) | |
| elif len(srids) > 1: | |
| # Inconsistent SRIDs detected | |
| raise ValueError("Geometries in the series have inconsistent SRIDs.") |
… geometries instead of EWKB (apache#2121) * Implement scalable workaround by using native sedona geometries instead of ewkb, and fixing up the tests * Support empty_list input and use first_valid_index instead of 0 for get_srid * Skip ALL geopandas tests for shapely version 1 (require get_srid method)
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Geopandas.GeoSeries: Implement scalable workaround for GeoSeries.__init__() #2095What changes were proposed in this PR?
ps.Seriesfromps.Seriesinstead of callingto_pandas()get_srid()function in 2.0+ is now necessary.How was this patch tested?
Ensured existing tests still pass. Also I fixed up the dataframe test_area test in the process (the logic was previously wrong).
Switching to using proper geometry serialization means we no longer handle the following edge cases. There are open PRs or issues for each of them
Did this PR include necessary documentation updates?