Skip to content

update(compose): migrate CB-Spider to PostgreSQL metadata backend - #2610

Merged
cb-github-robot merged 2 commits into
cloud-barista:mainfrom
yunkon-kim:260720-14
Jul 21, 2026
Merged

cb-github-robot merged 2 commits into
cloud-barista:mainfrom
yunkon-kim:260720-14

Conversation

@yunkon-kim

Copy link
Copy Markdown
Member
  • Add cb-spider-postgres service replacing SQLite meta_db
  • Update CB-Spider to v0.12.37 with PostgreSQL connection
  • Add PostgreSQL credentials to .env.example for CB-Spider
  • Fix backup/restore scripts SQL queries for PostgreSQL stats

I am opening this as a Draft PR to verify the following.
For more details, please refer to the changes.

Note - The test to create infrastructure on AWS in MapUI has passed.

@seokho-son
Is it okay to upgrade to Spider v0.12.37?

@powerkimhub
I suggest improving environment variables to prevent special character encoding errors.
Is this possible?

(As-Is)

- SPIDER_METADB_URL=postgresql://${SP_POSTGRES_USER:-spider}:${SP_POSTGRES_PASSWORD:-spider}@cb-spider-postgres:5432/${SP_POSTGRES_DB:-cb_spider}

(To-Be)

- SPIDER_POSTGRES_ENDPOINT=cb-spider-postgres:5432
- SPIDER_POSTGRES_DATABASE=${SP_POSTGRES_DB:-cb_spider}
- SPIDER_POSTGRES_USER=${SP_POSTGRES_USER:-spider}
- SPIDER_POSTGRES_PASSWORD=${SP_POSTGRES_PASSWORD:-spider}

@seokho-son
Please, review the script changes.

The updates are to resolve the following issue.

cb-tumblebug-postgres | 2026-07-21 02:11:14.343 UTC [70] FATAL: terminating connection due to administrator command
cb-tumblebug-postgres | 2026-07-21 02:11:14.343 UTC [120] FATAL: terminating connection due to administrator command
cb-tumblebug-postgres | 2026-07-21 02:11:14.344 UTC [118] FATAL: terminating connection due to administrator command
cb-tumblebug-postgres | 2026-07-21 02:11:14.343 UTC [119] FATAL: terminating connection due to administrator command
cb-tumblebug-postgres | 2026-07-21 02:11:14.442 UTC [56] LOG: checkpoint starting: immediate force wait
cb-tumblebug-postgres | 2026-07-21 02:11:14.506 UTC [56] LOG: checkpoint complete: wrote 43 buffers (0.3%); 1 WAL file(s) added, 0 removed, 0 recycled; write=0.006 s, sync=0.008 s, total=0.064 s; sync files=12, longest=0.003 s, average=0.001 s; distance=6228 kB, estimate=6228 kB; lsn=0/1F33D70, redo lsn=0/1F33D38
cb-tumblebug-postgres | 2026-07-21 02:11:26.653 UTC [186] ERROR: column "tablename" does not exist at character 30
cb-tumblebug-postgres | 2026-07-21 02:11:26.653 UTC [186] STATEMENT:
cb-tumblebug-postgres | SELECT
cb-tumblebug-postgres | schemaname,
cb-tumblebug-postgres | tablename,
cb-tumblebug-postgres | pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size;
cb-tumblebug-postgres | (SELECT COUNT(*) FROM pg_catalog.pg_class WHERE relname = tablename) AS exists
cb-tumblebug-postgres | FROM pg_stat_user_tables
cb-tumblebug-postgres | ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
cb-tumblebug-postgres |

- Add cb-spider-postgres service replacing SQLite meta_db
- Update CB-Spider to v0.12.37 with PostgreSQL connection
- Add PostgreSQL credentials to .env.example for CB-Spider
- Fix backup/restore scripts SQL queries for PostgreSQL stats

@seokho-son seokho-son left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor fix :)

Comment thread docker-compose.yaml Outdated
Comment on lines +210 to +212
build:
context: ../cb-mapui
dockerfile: Dockerfile

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
build:
context: ../cb-mapui
dockerfile: Dockerfile
# build:
# context: ../cb-mapui
# dockerfile: Dockerfile

@seokho-son
seokho-son marked this pull request as ready for review July 21, 2026 16:29
@seokho-son

Copy link
Copy Markdown
Member

/lgtm

@github-actions github-actions Bot added the lgtm This PR is acceptable by at least one reviewer label Jul 21, 2026
@seokho-son

Copy link
Copy Markdown
Member

/approve

@github-actions github-actions Bot added the approved This PR is approved and will be merged soon. label Jul 21, 2026
@cb-github-robot
cb-github-robot merged commit 76deba2 into cloud-barista:main Jul 21, 2026
1 of 2 checks passed
@powerkimhub

Copy link
Copy Markdown
Member

@yunkon-kim @seokho-son

From Spider v0.12.38, MetaDB can be configured in three modes.
Please refer to the following options and guides.

  • Default Mode: SQLite
  • PostgreSQL URL Mode
  • PostgreSQL Key-Value Mode

Reference guides:

@seokho-son

Copy link
Copy Markdown
Member

@powerkimhub FYI, Spider v0.12.38 seems not available yet. I am waiting for it.

@powerkimhub

Copy link
Copy Markdown
Member

@seokho-son

  • Now available. Thanks for waiting~!

@yunkon-kim
yunkon-kim deleted the 260720-14 branch August 24, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved This PR is approved and will be merged soon. lgtm This PR is acceptable by at least one reviewer script

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants