Public dashboard showing community activity metrics for Apache projects, with 12-month trend lines for meaningful context.
Status: Working dashboard deployed at https://projects.apache.org/metrics/ β fetches foundation data from projects.apache.org, mailing list stats from Pony Mail Foal API, Git/GitHub activity (with SVN support), new committer dates from Whimsy LDAP, and renders a per-project dashboard with trend lines, releases, community growth, and activity trend classifications.
cd /path/to/comdev-metrics
cp config.example.yml config.yml
cp .secrets.example .secrets
# Edit .secrets with your GitHub token
# Edit config.yml as needed
# Run with uv (no install needed)
uv run asfmetrics
# Single project
uv run asfmetrics --project comdev
# Skip mailing lists (just fetch foundation + git data)
uv run asfmetrics --skip-mailing-lists
# Skip git/VCS collection
uv run asfmetrics --skip-git
# Force re-fetch everything (bypass cache)
uv run asfmetrics --force-refresh
# Re-fetch the GitHub repo inventory
uv run asfmetrics --refresh-repos
# View results
cd site && python3 -m http.server 8888
# Open http://localhost:8888# Install dev dependencies (includes pytest)
uv sync --all-groups
# Run all tests
uv run pytest -v- Fetches foundation-wide data from projects.apache.org JSON files (committees, rosters, podlings, repositories) and releases from the ASF release catalog
- Detects new committers via Whimsy LDAP
createTimestamp, cross-referenced with project group membership - Detects roster changes (new committers/PMC members) by diffing committee data between runs
- Discovers all mailing lists per project via Pony Mail Foal
preferences.json - Collects per-list message stats for a 12-month rolling window (with aggressive caching β past months are never re-fetched)
- Collects Git/VCS activity per project (commits, committers, PRs opened/merged) via GitHub API or SVN log
- Classifies project activity trends via deterministic quarter-over-quarter analysis (Sharp Decline / Declining / Dormant)
- Writes everything to a JSON data store (
site/data/) - Publishes a static HTML + JS dashboard with:
- Overview: community growth, releases, mailing list activity rankings, all projects table
- Activity Trends: projects flagged as declining or dormant, with methodology and thresholds
- Per-project pages: mailing lists, git repos, releases, community growth, trend badges
- Links to projects.apache.org for full project metadata (roster, repos, homepage)
Only lists with actual traffic in the 12-month window are shown.
Data is aggressively cached to minimize API calls:
- Past months are immutable. Once a month ends, its data can never change in the archive or git history. Cached data for completed months is never re-fetched.
- Only the current month is refreshed on each run (since it's still in progress).
- Same-day re-runs skip (cache hit) β no redundant API calls during testing.
- Cache locations:
- Mailing lists:
site/data/_cache/mailing_lists/<project>.json - Git activity:
site/data/_cache/git/<project>.json
- Mailing lists:
- Force refresh: Use
--force-refreshto clear all caches and re-fetch everything.
On a typical weekly run mid-month, the mailing list phase completes in seconds (cache hits from prior runs) instead of making hundreds of API calls.
See config.example.yml. Config is searched in order:
./config.yml~/.asfmetrics/config.yml/etc/asfmetrics/config.yml
GitHub token (needed for git activity collection; optional but strongly recommended):
GITHUB_TOKENenvironment variable.secretsfile in the project directory (line:GITHUB_TOKEN=ghp_...)~/.secretsfile (user-level fallback)config.ymldata_sources.git.tokenfield
Copy .secrets.example β .secrets and fill in your token.
The .secrets file is in .gitignore and will never be committed.
Most projects use GitHub. For projects still on Subversion, add an override
in config.yml:
project_overrides:
httpd:
vcs: svn
svn_path: https://svn.apache.org/repos/asf/httpd/httpd/trunkSVN-only projects can also be auto-detected from repositories.json β
if a project has SVN repos but no GitHub repos, the collector automatically
uses svn log.
See config.example.yml for full documentation of override options.
comdev-metrics/
βββ .secrets.example # Template for secrets (GitHub token)
βββ config.example.yml # Example configuration
βββ pyproject.toml # uv/hatch project definition
βββ src/asfmetrics/
β βββ __init__.py # Package version
β βββ cli.py # Entry point β orchestrates all phases
β βββ config.py # Config loading (YAML, multi-path lookup)
β βββ trends.py # Deterministic activity trend classification
β βββ collectors/
β β βββ cache.py # Shared caching utilities (load/save/invalidate)
β β βββ mailing_lists.py # Pony Mail Foal API + caching
β β βββ git_activity.py # GitHub API + SVN log collector + caching
β β βββ github_repos.py # Repo inventory + project classification
β β βββ projects_apache_org.py # Foundation JSON + Whimsy LDAP + roster differ
β βββ analysis/ # Trend calculations (placeholder)
β βββ output/
β βββ json_api.py # JSON data writer
βββ site/
β βββ index.html # Overview dashboard (tabs: All Projects, Growth, Releases, Mailing Lists, Activity Trends)
β βββ project.html # Per-project detail page (?id=projectname)
β βββ about.html # Data sources and methodology documentation
β βββ data/ # Generated JSON + cache (git-ignored)
β βββ _cache/
β β βββ doap-warnings.json
β β βββ mailing_lists/ # Per-project mailing list cache
β β βββ git/ # Per-project git activity cache
β β βββ committees.json
β β βββ releases.json
β β βββ new_committers.json
β β βββ mailing_summary.json
β β βββ activity_trends.json
β βββ _state/ # Roster diff state between runs
β βββ _project_map.json # GitHub repo β project classification
β βββ <project>.json # Mailing list data
β βββ <project>_git.json # Git activity data
βββ DATA_SOURCES.md # Where all the data comes from
βββ PLAN.md # Execution plan and milestones
βββ tests/ # Unit tests (pytest)
βββ LICENSE # Apache License 2.0
usage: asfmetrics [-h] [--config CONFIG] [--project PROJECT]
[--skip-mailing-lists] [--skip-git]
[--force-refresh] [--refresh-repos]
Options:
--config CONFIG Path to config.yml (default: auto-discover)
--project PROJECT Run for a single project only
--skip-mailing-lists Skip mailing list collection
--skip-git Skip git/VCS activity collection
--force-refresh Clear all caches and re-fetch everything
--refresh-repos Re-fetch the GitHub repo inventory (project β repos map)
The CLI runs these phases in order:
- Repo inventory β GitHub org listing β
_project_map.json(only when needed: first run,--refresh-repos, or--force-refresh) - Foundation data β projects.apache.org JSON + Whimsy LDAP new committer dates + roster diffing
- Mailing lists β per-project Pony Mail stats (skipped with
--skip-mailing-lists) + summary file - Git/VCS activity β per-project GitHub API or SVN log (skipped with
--skip-git) - Activity trend classification β deterministic QoQ trend analysis β
activity_trends.json
Five tabs:
- All Projects (default): alphabetical table of all active PMCs with established date, roster size, chair
- Community Growth: two-color bar chart (blue = new committers, orange = new PMC members)
- Recent Releases: ranked by release count
- Mailing Lists: all lists ranked by 12-month message volume
- Activity Trends: projects classified as Sharp Decline, Declining, or Dormant with methodology
- Stat cards: messages, active lists, PMC size, releases, established date
- Activity trend badge (if applicable)
- Mailing list table with sparklines β click to expand full bar chart with trend line
- Git repos table with sparklines β click to expand commit chart + PR chart
- Recent releases (collapsed after 10)
- Community growth: new committers with names and dates
- Link to projects.apache.org for full metadata
Explains all data sources, methodology, what's measured, and limitations.
Production:
The code and data are installed on the projects.apache.org host, at /var/www/metrics with output at https://projects.apache.org/metrics.
The build.sh script is run as a daily cron job:
Sundays, 19:45 ./build.sh weekly >>/var/log/www-data/metrics_$(date "+%Y-%m").log
Other days, 19:40 ./build.sh daily >>/var/log/www-data/metrics_$(date "+%Y-%m").log
Local testing:
https://apache.github.io/comdev-metrics/ β built and deployed via GitHub Actions.
The GitHub Actions workflow (.github/workflows/metrics.yml):
- Downloads the previously-deployed site (contains cached data from prior runs)
- Copies
config.example.ymlβconfig.yml - Runs the full collection pipeline (
uv run asfmetrics) - Uploads the entire
site/directory as a Pages artifact (14-day retention) - Deploys to GitHub Pages
No files in the repository are modified by the build β only the Pages artifact
is updated. The GitHub Actions-provided GITHUB_TOKEN handles API
authentication automatically (ASF has GitHub Enterprise rate limits).
Currently triggered manually (workflow_dispatch); scheduled runs TBD.
There is another workflow (.github/workflows/updatecode.yml) which can be manually triggered
to regenerate the site without downloading any Git or mail data.
For local testing:
cp config.example.yml config.yml
cp .secrets.example .secrets # add your GitHub token
uv run asfmetrics --project comdev
uv run pytest -v # run testsApache License 2.0