Skip to content

[GH-2095] Geopandas: Implement Scalable workaround and store as geometries instead of EWKB - #2121

Merged
jiayuasu merged 3 commits into
apache:masterfrom
petern48:scalable_workaround_no_ewkb
Jul 19, 2025
Merged

jiayuasu merged 3 commits into
apache:masterfrom
petern48:scalable_workaround_no_ewkb

Conversation

@petern48

@petern48 petern48 commented Jul 19, 2025 •

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

  • Implement scalable workaround by manually implementing logic for ps.Series from ps.Series instead of calling to_pandas()
  • Revert the changes from [GH-2079] Store geometries in EWKB format in spark df and enforce it #2080
    • It didn't work properly with the once we used proper ps.Series because the serialization format conflicted.
    • After digging into the C code, it turns out the serialization method does include the SRID information, I just needed to know how to access it.
  • Modify tests to avoid small edge cases that we no longer handle
  • Remove shapely 1.X support for geopandas because the get_srid() function in 2.0+ is now necessary.

How was this patch tested?

Ensured existing tests still pass. Also I fixed up the dataframe test_area test in the process (the logic was previously wrong).

Switching to using proper geometry serialization means we no longer handle the following edge cases. There are open PRs or issues for each of them

  • UDT arrays where the first element is None
  • LinearRing geometry type

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@petern48
petern48 force-pushed the scalable_workaround_no_ewkb branch from cc6484f to 7553142 Compare July 19, 2025 03:28
@petern48
petern48 force-pushed the scalable_workaround_no_ewkb branch from aaf3fb8 to cc5d946 Compare July 19, 2025 04:34
@petern48
petern48 requested a review from zhangfengcdt July 19, 2025 05:12
@petern48
petern48 marked this pull request as ready for review July 19, 2025 05:12
@petern48
petern48 requested a review from jiayuasu as a code owner July 19, 2025 05:12
@jiayuasu
jiayuasu requested a review from Copilot July 19, 2025 05:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements significant scalability improvements and changes to geopandas support by storing geometries natively instead of as EWKB format, requiring Shapely 2.0+, and fixing test cases to handle edge cases that are no longer supported.

  • Reimplemented GeoSeries construction to avoid calling to_pandas() for better scalability
  • Changed geometry storage from EWKB back to native geometry format to properly access SRID information
  • Added Shapely 2.0+ requirement and corresponding test skips for older versions

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
python/tests/geopandas/test_sjoin.py Added Shapely 2.0+ version requirement check
python/tests/geopandas/test_match_geopandas_series.py Added version check and updated test data to avoid edge cases
python/tests/geopandas/test_match_geopandas_dataframe.py Added Shapely 2.0+ version requirement check
python/tests/geopandas/test_geoseries.py Added version check and commented out LinearRing test cases
python/tests/geopandas/test_geodataframe.py Added version check and reorganized constructor tests
python/sedona/geopandas/geoseries.py Major refactoring to handle geometry types natively and implement scalable construction
python/sedona/geopandas/geodataframe.py Simplified area property and removed EWKB conversion logic
Comments suppressed due to low confidence (3)

python/tests/geopandas/test_match_geopandas_series.py:330

  • The test data was modified to avoid edge cases with arrays starting with None, but there's no accompanying test to verify that this edge case is properly handled or documented as unsupported.
        data = [Point(1, 1), None, None, None, Point(0, 1)]

python/tests/geopandas/test_geoseries.py:233

  • Similar to the other test file, the test data was simplified to avoid edge cases, but there's no test coverage to verify the behavior of the removed edge case.
        data = [Point(0, 0), None]

python/sedona/geopandas/geoseries.py:186

  • [nitpick] The error message could be more helpful by suggesting what types are expected or how to convert the data to a compatible format.
            raise TypeError(

LineString([(0, 0), (1, 1), (0, 0)]),
LinearRing([(0, 0), (1, 1), (1, 0), (0, 1), (0, 0)]),
LinearRing([(0, 0), (-1, 1), (-1, -1), (1, -1)]),
# Errors for LinearRing: issue #2120

Copilot AI Jul 19, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment references issue #2120 but doesn't explain why LinearRing causes errors or what the expected behavior should be. This makes it difficult for future maintainers to understand the limitation.

Suggested change
# Errors for LinearRing: issue #2120
# LinearRing geometries are excluded due to known issues with
# compatibility in GeoSeries.is_simple. Specifically, LinearRing
# objects can cause unexpected errors when processed. See issue #2120
# for more details. The expected behavior is that LinearRing objects
# should be handled similarly to LineString objects without errors.

Copilot uses AI. Check for mistakes.
Comment on lines +95 to +103
first_idx = ps_series.first_valid_index()
if first_idx is not None:
geom = ps_series.iloc[int(first_idx)]
srid = shapely.get_srid(geom)

# Shapely objects stored in the ps.Series retain their srid
# but the GeoSeries does not, so we manually re-set it here
if srid > 0:
result.set_crs(srid, inplace=True)

Copilot AI Jul 19, 2025

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code accesses the first valid geometry to extract SRID information, but doesn't handle the case where geometries in the same series might have different SRIDs, which could lead to inconsistent behavior.

Suggested change
first_idx = ps_series.first_valid_index()
if first_idx is not None:
geom = ps_series.iloc[int(first_idx)]
srid = shapely.get_srid(geom)
# Shapely objects stored in the ps.Series retain their srid
# but the GeoSeries does not, so we manually re-set it here
if srid > 0:
result.set_crs(srid, inplace=True)
valid_geometries = ps_series.dropna()
srids = {shapely.get_srid(geom) for geom in valid_geometries if shapely.get_srid(geom) > 0}
if len(srids) == 1:
# All SRIDs are consistent, set the CRS
result.set_crs(srids.pop(), inplace=True)
elif len(srids) > 1:
# Inconsistent SRIDs detected
raise ValueError("Geometries in the series have inconsistent SRIDs.")

Copilot uses AI. Check for mistakes.
@jiayuasu jiayuasu added this to the sedona-1.8.0 milestone Jul 19, 2025
@jiayuasu
jiayuasu merged commit f9ee970 into apache:master Jul 19, 2025
@petern48
petern48 deleted the scalable_workaround_no_ewkb branch July 19, 2025 14:05
Kontinuation pushed a commit to Kontinuation/sedona that referenced this pull request Jan 21, 2026
… geometries instead of EWKB (apache#2121)

* Implement scalable workaround by using native sedona geometries instead of ewkb, and fixing up the tests

* Support empty_list input and use first_valid_index instead of 0 for get_srid

* Skip ALL geopandas tests for shapely version 1 (require get_srid method)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Geopandas.GeoSeries: Implement scalable workaround for GeoSeries.__init__()

3 participants