Skip to content

Adding Mapping documents - #305

Open
grant-minor-sntialtech wants to merge 4 commits into
mainfrom
Mapping-Docs
Open

grant-minor-sntialtech wants to merge 4 commits into
mainfrom
Mapping-Docs

Conversation

@grant-minor-sntialtech

Copy link
Copy Markdown
Contributor

There are normal and 'verbose' versions here. Verbose is meant for ai ingestion

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several mappings, source references, and analysis counts remain inconsistent or unsupported.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds paired summary and verbose source-to-PDC mapping analyses for TechSoup OKF, Grantmakers.io, Cuoree Mani, and Charity Navigator.

Changes:

  • Documents field coverage, gaps, and data-quality concerns.
  • Maps external fields to PDC fields and notes modeling limitations.
  • Provides detailed analyses intended for AI ingestion.
File summaries
File Description
docs/mappings/techsoup-okf-field-gap-analysis.verbose.md Detailed TechSoup analysis
docs/mappings/techsoup-okf-field-gap-analysis.md TechSoup summary
docs/mappings/grantmakersIO-field-mapping.verbose.md Detailed Grantmakers.io analysis
docs/mappings/grantmakersIO-field-mapping.md Grantmakers.io summary
docs/mappings/CuoreeMani-field-mapping.verbose.md Detailed Cuoree Mani mapping
docs/mappings/CuoreeMani-field-mapping.md Cuoree Mani summary
docs/mappings/charityNavigator-field-gap-analysis.verbose.md Detailed Charity Navigator analysis
docs/mappings/charityNavigator-field-gap-analysis.md Charity Navigator summary
Review details

Suppressed comments (23)

docs/mappings/CuoreeMani-field-mapping.md:31

  • The summary repeats the same nonexistent/uncorroborated organization_fiscal_year target. Use the existing fiscal-year-end-year base field (or mark the year as having no target) so the summary does not advertise an invalid mapping.
| 13  | Current Fiscal Year Budget | `organization_operating_budget` (pair with `organization_fiscal_year`)            | organization  | ✅ Close      |

docs/mappings/CuoreeMani-field-mapping.verbose.md:20

  • organization_fiscal_year is not referenced elsewhere as a PDC base field, while the catalog mapping in grantmakersIO-field-mapping.md:110 uses organization_fiscal_year_end_date_year. As written, this presents a likely nonexistent target as part of an existing-field mapping.
| 13  | Current Fiscal Year Budget | `organization_operating_budget`<br>(pair with `organization_fiscal_year` to state which year) | organization           | string | ✅ Close      |

docs/mappings/charityNavigator-field-gap-analysis.verbose.md:21

  • This source citation has the same broken relative path from docs/mappings/; the reference should climb two directories and retain the line anchor.
From `baseFieldMap` in [`src/charityNavigator.ts:44`](src/charityNavigator.ts):

docs/mappings/charityNavigator-field-gap-analysis.verbose.md:36

  • This query citation also resolves under docs/mappings/src/ and is therefore broken. Point it at the actual source file with a repository-relative path.
These fields are already requested by the `NonprofitsPublic` GraphQL query ([`src/charityNavigator.ts:74`](src/charityNavigator.ts)) but are dropped before writing to PDC.

docs/mappings/grantmakersIO-field-mapping.md:138

  • These are the foundation's own charitable activities and expenses, not a changemaker's applicant proposal. Mapping them to proposal_* fields would create a misleading proposal-shaped record; mark them as no target or specify a separate funder-program model.
| `charitable_activities[].description`                      | `proposal_related_programs`, `proposal_activities`                          |
| `charitable_activities[].expenses`                         | `proposal_budget` / `proposal_project_budget`                               |

docs/mappings/grantmakersIO-field-mapping.md:125

  • people[] is not a board-only list; it also supplies executive and staff mappings below. Mapping the full array to board-member names conflates those roles and would overstate board membership. Filter to director/trustee titles first.
| `people[]`                                   | `organization_board_members_names` (file), `organization_board_members_count`, `organization_leadership` |

docs/mappings/grantmakersIO-field-mapping.md:91

  • The summary repeats the same boolean/value confusion: website_is_an_email = true identifies the kind of the source value; it is not the value that belongs in organization_email. Identify and normalize the underlying email value instead.
| `website_is_an_email = true`                                | `organization_email` — **prevents writing a `mailto:` into the website field**                     |

docs/mappings/grantmakersIO-field-mapping.md:90

  • This summary mapping omits the required type gate: contact.website.cleaned can represent an email, phone, unknown value, or URL. It should populate organization_website only when the detected type is URL and the status is accepted; otherwise route it to the appropriate field or skip it.
| `contact.website.cleaned`                                   | `organization_website` — **use `cleaned`, not `website_verbatim`**                                 |

docs/mappings/grantmakersIO-field-mapping.md:114

  • eobmf_ruling_date is only yyyymm in the source, but this row presents a direct mapping to a date field. Converting it to a full date would invent day precision; document a supported month-precision representation or keep it in a non-date field/provenance instead.
| `eobmf_ruling_date` (yyyymm)                   | `organization_irs_ruling_date` (date) / `organization_ruling_date`   |

docs/mappings/grantmakersIO-field-mapping.md:163

  • INCOME_AMT/REVENUE_AMT are explicitly described as banded source values, not exact currency amounts, so they cannot be written directly to PDC currency fields; INCOME_AMT also is not net income because no expenses are available. Leave these targets empty or define an explicit band representation instead of presenting this as a usable mapping.
| `ASSET_AMT` / `INCOME_AMT` / `REVENUE_AMT`     | `organization_total_assets` / `_net_income` / `_total_revenue` (banded)            |

docs/mappings/grantmakersIO-field-mapping.verbose.md:18

  • eobmf_ruling_date is a date string in the referenced GrantmakersExtractedDataObj; it is not a boolean. Calling both surfaced values “derived booleans” misdescribes the available upstream data and can lead readers to ignore the ruling-date field.
- **The largest untapped org-level source is the EOBMF join the pipeline already performs.** `shared/typings/irs/all.ts` shows grantmakers.io ingests the full IRS Exempt Organizations Business Master File (`NTEE_CD`, `SUBSECTION`, `CLASSIFICATION`, `FOUNDATION`, `DEDUCTIBILITY`, `RULING`, `STATUS`, addresses, financial bands) but surfaces only **two** derived booleans from it. PDC has exact base-field targets for **~15 more** of those columns — including `organization_ntee_code`, which PDC wants and grantmakers.io already holds.

docs/mappings/grantmakersIO-field-mapping.verbose.md:207

  • charitable_activities describes a foundation's own direct activities, not an applicant proposal. Writing it to proposal_*/proposal_budget on the foundation's changemaker would misrepresent the foundation as a proposal applicant; leave it unmapped or define an explicit funder-program object.
| `charitable_activities[].description`             | `proposal_related_programs`, `proposal_activities`                                                                                                           | ✅ High — 990-PF Part IX-A direct charitable activities  |
| `charitable_activities[].expenses`                | `proposal_budget` / `proposal_project_budget`                                                                                                                | ⚠️ Medium                                                |

docs/mappings/grantmakersIO-field-mapping.verbose.md:190

  • The full people[] array includes officers/key employees as well as directors/trustees; the same table uses it to derive executives and paid staff. It therefore cannot populate board-member names/count without role filtering. Restrict this mapping to board roles or mark it unavailable.
| `people[]` (full array)                                  | `organization_board_members_names` (`file`)                                           | ✅ High                                                       |

docs/mappings/grantmakersIO-field-mapping.verbose.md:147

  • contact.website.status !== 'INVALID' is not enough to validate a website: the upstream type also allows EMAIL, PHONE-US, and UNKNOWN, and NEEDS_REVIEW is not a confirmed URL. Gate organization_website on a URL type plus an accepted status, and route or skip non-URL values.
| `contact.website.status` (`VALID`/`REPAIRED`/`INVALID`/`NEEDS_REVIEW`), `.type`, `.notes[]` | _(provenance — gate ingest on `status !== 'INVALID'`)_    | ➖                                                                                |

docs/mappings/grantmakersIO-field-mapping.verbose.md:148

  • website_is_an_email is only a boolean flag, not the email address itself. Mapping this row directly leaves no value to write to organization_email; use the underlying website/contact.website value when its detected type is EMAIL, normalize it to an address, and do not emit the boolean or a mailto: URI.
| `website_is_an_email` = `true`                                                              | `organization_email` (the value is an address, not a URL) | ✅ High — **important**: prevents writing a `mailto:` into `organization_website` |

docs/mappings/grantmakersIO-field-mapping.verbose.md:255

  • This mapping would misstate the EOBMF data: the row labels INCOME_AMT and REVENUE_AMT as banded, while organization_net_income and the other targets are exact currency concepts. A band cannot be emitted as a currency value, and net income cannot be derived without expenses; keep these fields unmapped or model the band explicitly.
| `ASSET_AMT` / `INCOME_AMT` / `REVENUE_AMT`          | `organization_total_assets` / `organization_net_income` / `organization_total_revenue`                   | Medium — banded                                        |

docs/mappings/techsoup-okf-field-gap-analysis.md:84

  • The verbose analysis says four mapped values are metadata in §2 and §5, but this summary calls them three. That makes the stated data-quality count internally inconsistent; it should say four (or identify a different, consistent subset).
2. **`proposal_name` names the wrong organization** ("Cumberland Gap Health Cooperative" on the Black Mountain record). Upstream content bug, propagated verbatim. Add an ingest assertion.

docs/mappings/techsoup-okf-field-gap-analysis.md:46

  • The README title is an organization/display name, not evidence of a legal name, and it is already used for organization_name. This summary should not claim a separate organization_legal_name mapping without an explicitly labeled legal-name source.
| README title                                            | `organization_legal_name`                                                                               |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:157

  • This row has only two cells even though the table declares Bundle data, Value, and PDC base field columns, so the target is rendered under the wrong column and the source value is missing. Split the screening label from its all cleared value.
| Screening cleared: OFAC, PEP, debarment, adverse media, FATF, sanctions                                             | `steps_to_prevent_illegal_activity` (imperfect — see §7)     |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:64

  • README.md#title supplies the bundle's display/organization name, but it does not identify a legal name; that title is already mapped to organization_name. Treating it as organization_legal_name asserts an unverified legal fact.
| Legal name                | synthetic-Black Mountain Workforce Partnership                                 | `organization_legal_name`        | `README.md#title`                       |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:76

  • These sources provide only a year (2013), while the proposed ruling/start-date targets are date fields. Emitting a made-up month/day would lose precision; mark these as year-only or leave the date targets empty until the source provides a full date.
| Founded               | 2013                       | `organization_start_date`                                                                                  | `README.md` body                  |
| IRS exemption granted | 2013 ("the same year")     | `organization_ruling_date` / `organization_irs_ruling_date`                                                | `README.md` body                  |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:180

  • The source gives only a relative 5–8 week window, not a calendar start date. Mapping this row to proposal_start_date would create a false date; keep only proposal_duration until an actual start date is supplied.
| 5–8 weeks, in a between-cohort window                                                 | `proposal_duration`, `proposal_start_date`                                              |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:16

  • This says three narrative fields, but §2 and §5.3 enumerate four: proposal_project_outcomes, community_definition_overview, proposal_related_programs, and proposal_funding_summary. Please correct the count so the executive summary matches the detailed analysis.
- **Three genuine data-quality bugs** in the current 14 mappings should be fixed regardless of scope: `organization_status` carries a _document_ lifecycle value, `proposal_name` names the **wrong organization**, and three narrative fields carry the file's meta-disclaimer instead of its content. See §5.
  • Files reviewed: 8/8 changed files
  • Comments generated: 10
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/mappings/charityNavigator-field-gap-analysis.verbose.md Outdated

| Enhancement | PDC base field |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`grantee_ein`** | `organization_tax_id` (recipient) ⭐⭐ **the missing join key** — LLM-derived with `confidence` + `EnhancementSource`; accept only `llm_confirmed` / `direct_match` / `high` |
- **`goodAsOf` from `last_updated_irs`, not `last_updated_grantmakers`** (the latter is a processing timestamp). Given the 9–18mo lag, many values arrive near or past their `valueRelevanceHours` window — a correct and useful signal, if the timestamp is honest.
- **`people[]` are named individuals with compensation.** Public on the 990-PF, but ingest at PDC's declared sensitivity (230 of 282 fields are `restricted`), not the IRS source's.
- **Consent posture differs** — this is data orgs filed under legal obligation, not chose to share. `pdc_share_data` should carry an explicit determination.
- **GivingTuesday overlap** — their repo carries a `GivTuesGrant` interface; this repo already ingests GT ([`src/getMetrics.ts`](../../src/getMetrics.ts)). Same underlying 990-PF filings, two providers → **guaranteed double-counting** unless dedup/precedence is settled first.

| Enhancement | PDC base field | Why it matters |
| -------------------------------------------------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `grantee_ein` | `organization_tax_id` (recipient) | ⭐⭐ **The missing join key.** LLM-derived with a `confidence` of `high`/`medium`/`low` and an `EnhancementSource` (`llm`, `llm_confirmed`, `direct_name_match`, `normalized_name_match`). PDC should accept only `llm_confirmed` / `direct_match` / `high`. |

### 6.4 Cross-reference: GivingTuesday

`shared/typings/grantmakers/all.ts` also defines a `GivTuesGrant` interface over the GivingTuesday 990-PF data mart (`FILEREIN`, `SIGOCPYRBNBN1`, `SIGOCPYAMOUN`, `SIGOCPYPOGOC`, …). Since this repo already ingests GivingTuesday ([`src/getMetrics.ts`](src/getMetrics.ts), branch `ENH-296`), **the same underlying 990-PF grants are reachable from two providers.** Deduplication strategy and provider precedence should be settled before adding grantmakers.io grants, or PDC will double-count.
## Gotchas / data-quality bugs

1. **`organization_status` = `"stable"` is the document lifecycle value**, not an org status — and PDC marks that field `public`. The real status is `ELIGIBLE` in `verification.md` → belongs in `review_status`.
2. **`proposal_name` names the wrong organization** ("Cumberland Gap Health Cooperative" on the Black Mountain record). Upstream content bug, propagated verbatim. Add an ingest assertion.
| Sustain/expand community visits; recruitment; general operating support | `proposal_activities`, `proposal_funding_category` |
| "Neither of us could run our half without the other" | `proposal_related_programs`, `significant_other_funders` |

⚠️ **This file's body is the wrong organization's content** — it describes a clinic with a nurse practitioner and a truck, and its `title` names "synthetic-Cumberland Gap Health Cooperative". That title is what currently populates `proposal_name`. See §5.

3. **Four values are metadata, not content.** `proposal_project_outcomes`, `community_definition_overview`, `proposal_related_programs`, and `proposal_funding_summary` all carry the file's authorial note ("Fabricated.", "Illustrative only; not part of the published civic/0.6 profile.") rather than the file's substance.

4. **Everything is typed `"string"`.** PDC declares 7 `currency`, 5 `email`, 4 `date`, 1 `number`, 1 `phone_number`, and 8 `file` fields. If financials and dates are added per §3, `organization_total_revenue` / `organization_total_assets` must be emitted as `currency`, `organization_irs_ruling_date` as `date`, and `organization_volunteers` as `number`.
| **Total achievable with no schema change** | **~69** | **24.5%** |
| Requires new base fields (§7) | ~11 concepts | — |

By PDC category, the achievable additions land as: `organization` +22, `project` +20, `evaluation` +9, `outcomes` +3, `budget` +4, `methodology` +5, `partnerships` +1, `sustainability` +1.

## 9. Recommended order of work

1. **Fix the six data-quality defects in §5** — smallest effort, highest correctness return. In particular stop mapping the document `status` to `organization_status`, and add `goodAsOf` from `generated.at` / `sources[].last_modified` / `verified.at`.
@bickelj

bickelj commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

I am still not sure these docs belong here, but the worst that can happen is that they, like all other documentation, get stale.

Copilot had some suggestions (more than I expected)!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Broken references and unresolved mapping, schema, privacy, and data-quality inconsistencies remain.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (33)

docs/mappings/CuoreeMani-field-mapping.md:31

  • The summary repeats the undefined organization_fiscal_year code. Use the existing organization_fiscal_year_end_date_year field name, or omit the pairing until the target catalog confirms a different field.
| 13  | Current Fiscal Year Budget | `organization_operating_budget` (pair with `organization_fiscal_year`)            | organization  | ✅ Close      |

docs/mappings/CuoreeMani-field-mapping.verbose.md:20

  • organization_fiscal_year does not appear elsewhere in the repository's PDC field mappings; the existing field name for a year is organization_fiscal_year_end_date_year (used in the Grantmakers mapping). Recommending the former leaves consumers looking for a base field that is not defined here.
| 13  | Current Fiscal Year Budget | `organization_operating_budget`<br>(pair with `organization_fiscal_year` to state which year) | organization           | string | ✅ Close      |

docs/mappings/charityNavigator-field-gap-analysis.verbose.md:3

  • This relative link is resolved from docs/mappings, so src/charityNavigator.ts points at docs/mappings/src/charityNavigator.ts, not the repository's src/charityNavigator.ts. Use the repository-relative ../../src/charityNavigator.ts path.
**Subject:** [`src/charityNavigator.ts`](src/charityNavigator.ts) field mapping (`baseFieldMap`)

docs/mappings/charityNavigator-field-gap-analysis.verbose.md:21

  • This source link has the same broken base path: from docs/mappings, src/charityNavigator.ts:44 resolves under docs/mappings/src. Point it to the actual source file (ideally with a line anchor).
From `baseFieldMap` in [`src/charityNavigator.ts:44`](src/charityNavigator.ts):

docs/mappings/charityNavigator-field-gap-analysis.verbose.md:36

  • This link also resolves to a nonexistent docs/mappings/src directory. The query is in the repository root's src directory, so the link needs ../../src/charityNavigator.ts#L74.
These fields are already requested by the `NonprofitsPublic` GraphQL query ([`src/charityNavigator.ts:74`](src/charityNavigator.ts)) but are dropped before writing to PDC.

docs/mappings/chicago_partner_org-field-mapping.md:122

  • The summary has the same problem: school_years.number is not itself a date suitable for goodAsOf. Specify the date derivation or preserve the school-year period only as provenance.
- **`goodAsOf` from the school year, not `updated_at`.** A 2019 profile edited in 2026 is still a 2019 fact. Use `school_years.number`; keep `updated_at` as batch metadata.

docs/mappings/chicago_partner_org-field-mapping.md:42

  • This summary lists employees_count as a core organization_paid_staff mapping even though the Gotchas section says it is a portal-login count and should otherwise be left empty. Remove it from the reachable mapping (and coverage totals) or clearly mark it as excluded to avoid publishing false staff counts.
| `employees_count`                                 | `organization_paid_staff`                                        | ⚠️ **counts portal logins, not staff** |

docs/mappings/chicago_partner_org-field-mapping.verbose.md:228

  • A school-year label such as 2024–25 is a period, not a point-in-time date. Deriving goodAsOf directly from school_years.number can create a fabricated freshness date; use an explicit period start/end date if the schema provides one, otherwise keep the school year in provenance and do not synthesize goodAsOf.
6. **`goodAsOf` must come from the school year, not `updated_at`.** `updated_at` records when a row was last touched in the application; `school_year_id` records what period the value describes. A 2019 profile edited in 2026 is still a 2019 fact. Resolve `school_year_id` → `chicago.school_years.number` and derive `goodAsOf` from that, using `updated_at` only as batch metadata.

docs/mappings/chicago_partner_org-field-mapping.verbose.md:270

  • §3.2 presents district_funding_cents as mappable to significant_other_funders, but this section calls the same value "no PDC target". Distinguish an exact target from the lossy interim fallback so an implementer does not treat a single funding stream as a general partnerships fact.
| **District funding received** (`district_funding_cents`)                                     | Revenue from one named public source. `organization_total_revenue` would misstate it as a total                                                  | `organization_funding_by_source`, or a repeatable funder/amount pair   |

docs/mappings/chicago_partner_org-field-mapping.verbose.md:3

  • The only schema source is identified by a contributor's workstation-local path, so readers and CI cannot follow or reproduce this analysis from a checkout. Replace it with a repository-relative or stable external reference, or include the workbook as a reviewable artifact.
**Subject:** `C:\Data\Parliament\chicago_partner_org_metadata.xlsx` — the `columns` tab (schema inventory) and the `relationships` tab (join contract)

docs/mappings/chicago_partner_org-field-mapping.verbose.md:326

  • Filtering details.state = 'approved' here discards every archived school-year row, even though the mapping earlier treats details.state as the source for organization_status (archived = inactive) and recommends one value per school year. Either retain both states for historical status values or explicitly scope the feed to active years and remove the archived-status expectation.
5. **Apply the §8 gates in the extract query**, not downstream — `restrict_access`, `mask_pii`, `details.state = 'approved'`, `allocations.state` approved, and `*_type = 'Organization'` on every polymorphic join.

docs/mappings/chicago_partner_org-field-mapping.verbose.md:324

  • The source description says the EIN is nullable and recommends measuring its fill rate, but this step unconditionally proposes one changemaker per organization keyed on EIN. Organizations without an EIN need a stable artlook id fallback or an explicit exclusion rule; otherwise this implementation plan drops part of the source.
3. **Ship the organization core first** — §3.1 identity + §3.2 profile + §3.3 address. Roughly 25 fields, high confidence, one changemaker per `chicago.organizations` row keyed on EIN.

docs/mappings/chicago_partner_org-field-mapping.verbose.md:93

  • This is listed as a mapping to organization_paid_staff, but §5.2 says the value counts portal accounts rather than staff and recommends leaving the field empty. Keeping it in the organization mapping can still lead to false headcount data; mark it as excluded rather than reachable.
| `employees_count`                | `organization_paid_staff`                  | string   | ⚠️ **Low**  | Counts _portal users affiliated with the org_, not headcount — see §5                   |

docs/mappings/grantmakersIO-field-mapping.md:60

  • The summary still recommends mapping the filing year directly to proposal_date, which would produce an invalid or misleading date value. Treat tax_year as provenance only and derive goodAsOf from last_updated_irs.
| `tax_year`                          | `proposal_date` + drives `goodAsOf`                                         | ⚠️ filing year ≠ award date               |

docs/mappings/grantmakersIO-field-mapping.md:125

  • The summary repeats the same over-broad mapping: people[] is also used to derive executives, so its full length cannot be treated as board membership. Use only records identified as directors/trustees for the board fields.
| `people[]`                                   | `organization_board_members_names` (file), `organization_board_members_count`, `organization_leadership` |

docs/mappings/grantmakersIO-field-mapping.md:138

  • The summary presents ongoing foundation activities as proposal fields without the scope caveat in the verbose analysis. This can cause consumers to interpret Part IX-A activities as grant applications or funded projects; label it as an explicitly lossy fallback.
| `charitable_activities[].description`                      | `proposal_related_programs`, `proposal_activities`                          |
| `charitable_activities[].expenses`                         | `proposal_budget` / `proposal_project_budget`                               |

docs/mappings/grantmakersIO-field-mapping.md:171

  • The summary uses direct_match, but the source enum listed in the verbose mapping is direct_name_match. This token mismatch makes the proposed acceptance gate ambiguous or impossible to implement.
| **`grantee_ein`**                            | `organization_tax_id` (recipient) ⭐⭐ **the missing join key** — LLM-derived with `confidence` + `EnhancementSource`; accept only `llm_confirmed` / `direct_match` / `high` |

docs/mappings/grantmakersIO-field-mapping.md:192

  • This link points to getMetrics.ts, which only reports endpoint counts; it is not the GivingTuesday importer. Link to src/givingTuesday.ts instead, otherwise readers are sent to unrelated code when following the deduplication warning.
- **GivingTuesday overlap** — their repo carries a `GivTuesGrant` interface; this repo already ingests GT ([`src/getMetrics.ts`](../../src/getMetrics.ts)). Same underlying 990-PF filings, two providers → **guaranteed double-counting** unless dedup/precedence is settled first.

docs/mappings/grantmakersIO-field-mapping.md:125

  • The summary omits the required conversion for organization_board_members_names: people[] is structured data and the target is a file field, not a directly writable string/array. Add the serialization/upload requirement so this does not get implemented as a direct mapping.
| `people[]`                                   | `organization_board_members_names` (file), `organization_board_members_count`, `organization_leadership` |

docs/mappings/grantmakersIO-field-mapping.verbose.md:85

  • tax_year is only a filing year, not a grant/proposal date, and the target is a date field. Writing a bare number such as 2023 to proposal_date would either be invalid or falsely imply an award date. Keep the year as provenance and use the IRS source timestamp for goodAsOf unless an actual award date is available.
| `tax_year`                   | number         | `proposal_date`; also drives `goodAsOf`                                                                        | project    | ⚠️ Filing year, not award date                                                           |

docs/mappings/grantmakersIO-field-mapping.verbose.md:191

  • The full people[] array also supplies the executive-administrator rows immediately below, so it is not a board-only collection. Mapping the whole array and its length to board names/count will include officers and other listed people; filter to director/trustee roles before populating these board fields.
| `people[]` (full array)                                  | `organization_board_members_names` (`file`)                                           | ✅ High                                                       |
| `people[].length`                                        | `organization_board_members_count`                                                    | ✅ High                                                       |

docs/mappings/grantmakersIO-field-mapping.verbose.md:195

  • people[].compensation includes the same mixed population used for executive extraction, so it is not automatically governing-body compensation. Without filtering board/trustee roles, this mapping can publish officer or staff pay as board compensation.
| `people[].compensation` (+ `hours`)                      | `organization_governing_body_compensation`                                            | ✅ High — direct match, and rarely available elsewhere        |

docs/mappings/grantmakersIO-field-mapping.verbose.md:267

  • direct_match is not one of the EnhancementSource values enumerated in this same row (direct_name_match is). As written, an implementer could gate out the actual direct-name matches; use the exact enum name and keep the source/confidence checks distinct.
| `grantee_ein`                                                                                      | `organization_tax_id` (recipient)            | ⭐⭐ **The missing join key.** LLM-derived with a `confidence` of `high`/`medium`/`low` and an `EnhancementSource` (`llm`, `llm_confirmed`, `direct_name_match`, `normalized_name_match`). PDC should accept only `llm_confirmed` / `direct_match` / `high`. |

docs/mappings/grantmakersIO-field-mapping.verbose.md:207

  • These are the foundation's ongoing Part IX-A charitable activities, not proposal submissions. Writing them to proposal_* fields with high confidence makes them look like funded proposals; mark this as a lossy fallback (or keep it organization-scoped) and lower the confidence.
| `charitable_activities[].description`             | `proposal_related_programs`, `proposal_activities`                                                                                                           | ✅ High — 990-PF Part IX-A direct charitable activities  |
| `charitable_activities[].expenses`                | `proposal_budget` / `proposal_project_budget`                                                                                                                | ⚠️ Medium                                                |

docs/mappings/grantmakersIO-field-mapping.verbose.md:294

  • This getMetrics.ts link is resolved relative to docs/mappings, so it points at docs/mappings/src/getMetrics.ts, which does not exist. Use the repository-relative ../../src/getMetrics.ts path.
`shared/typings/grantmakers/all.ts` also defines a `GivTuesGrant` interface over the GivingTuesday 990-PF data mart (`FILEREIN`, `SIGOCPYRBNBN1`, `SIGOCPYAMOUN`, `SIGOCPYPOGOC`, …). Since this repo already ingests GivingTuesday ([`src/getMetrics.ts`](src/getMetrics.ts), branch `ENH-296`), **the same underlying 990-PF grants are reachable from two providers.** Deduplication strategy and provider precedence should be settled before adding grantmakers.io grants, or PDC will double-count.

docs/mappings/grantmakersIO-field-mapping.verbose.md:196

  • people[].hours > 0 does not identify employees or paid staff: the available person records include board/officer people, and the typings expose hours and compensation but no staff classification. Mapping that count to organization_paid_staff can materially overstate staffing; leave the field unmapped or document a separately validated proxy.
| `people[]` where `hours > 0` (count)                     | `organization_paid_staff`                                                             | ⚠️ Medium — better than the boolean below                     |

docs/mappings/grantmakersIO-field-mapping.verbose.md:190

  • people[] is a structured array, while organization_board_members_names is documented here as a PDC file field. The array cannot be written directly as that field value; document the required file serialization/upload or use a compatible text representation before calling this a high-confidence direct mapping.
| `people[]` (full array)                                  | `organization_board_members_names` (`file`)                                           | ✅ High                                                       |

docs/mappings/techsoup-okf-field-gap-analysis.md:63

  • This row presents the funding file as mappable, but the file is explicitly titled and written for Cumberland Gap rather than Black Mountain. An implementer following this summary could attach the clinic's funding request to the Black Mountain record; mark it blocked until the source is reassigned or corrected.
| `what_i_need_funding_for.md` body                  | `proposal_funding_summary`, `proposal_need`, `proposal_purpose_statement`, `proposal_activities`, `proposal_funding_category`                                                                                                                                                                                   |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:257

  • These category counts sum to 65 (22 + 20 + 9 + 3 + 4 + 5 + 1 + 1), but §8 says there are only ~55 additional fields and ~69 total. Since each base field belongs to one category, reconcile this breakdown with the coverage table before using it for prioritization.
By PDC category, the achievable additions land as: `organization` +22, `project` +20, `evaluation` +9, `outcomes` +3, `budget` +4, `methodology` +5, `partnerships` +1, `sustainability` +1.

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:263

  • §5 contains seven numbered data-quality issues, so this work item is off by one and makes the recommended order internally inconsistent. Either say seven or avoid hard-coding the count.
1. **Fix the six data-quality defects in §5** — smallest effort, highest correctness return. In particular stop mapping the document `status` to `organization_status`, and add `goodAsOf` from `generated.at` / `sources[].last_modified` / `verified.at`.

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:16

  • This bullet says three narrative fields carry the metadata disclaimer, but §5.3 identifies four: proposal_project_outcomes, community_definition_overview, proposal_related_programs, and proposal_funding_summary. Please make the count consistent with the detailed list.
- **Three genuine data-quality bugs** in the current 14 mappings should be fixed regardless of scope: `organization_status` carries a _document_ lifecycle value, `proposal_name` names the **wrong organization**, and three narrative fields carry the file's meta-disclaimer instead of its content. See §5.

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:157

  • §7 correctly says the screening battery has no existing PDC target because steps_to_prevent_illegal_activity describes prevention methodology, not screening results. Listing it here as an evaluation mapping both suggests a semantically wrong field and inflates the existing-field coverage; remove it from the mappable table or explicitly mark it as a lossy interim workaround and recalculate the totals.
| Screening cleared: OFAC, PEP, debarment, adverse media, FATF, sanctions                                             | `steps_to_prevent_illegal_activity` (imperfect — see §7)     |

docs/mappings/techsoup-okf-field-gap-analysis.verbose.md:224

  • §5.2 says this file's title and body are actually the Cumberland Gap clinic's content. Emitting it as a proposal against the Black Mountain changemaker would still publish another organization's funding request on the wrong record; it must be reassigned or excluded before the two-proposal recommendation is applied.
**Recommendation:** emit these as **two separate proposals** against the same changemaker before expanding proposal-side coverage. If the target schema cannot express that yet, map only the funding need to `proposal_*` fields and keep the volunteer request in organization-scoped fields (`organization_workplace_issues`) until it can.
  • Files reviewed: 10/10 changed files
  • Comments generated: 11
  • Review effort level: Lite

Comment on lines +26 to +28
## Phase 1 — add now (target exists, currently empty)

Requires adding these to the GraphQL query + interface + `baseFieldMap`. No PDC schema change.
Comment on lines +59 to +61
Charity Navigator's public nonprofit type exposes more than the 12 fields the current query selects. Per Charity Navigator's own [`cn-examples`](https://github.com/CharityNavigator/cn-examples) query samples, the public nonprofit object also includes a **structured address** and organization URLs. These have **ready, currently-empty PDC targets**:

| CN attribute (available, not queried) | Recommended PDC base field | PDC data type | Notes |
| | |
| ---------------------- | ----------------------------------------------------------------------------------------- |
| **Source** | artlook Chicago — arts-education partner portal (Rails/Postgres, multi-tenant) |
| **Schema doc** | `C:\Data\Parliament\chicago_partner_org_metadata.xlsx` — `columns` + `relationships` tabs |

## 🚩 Read first

1. **`chicago.*` is a strict subset of `public.*`.** `public` is the multi-tenancy _template_, not Chicago data. Read entities from `chicago.*`, shared lookups (`users`, `positions`, `grades`, `program_types`, `communities`) from `public.*`. **Ingesting both duplicates every changemaker.**
# Chicago Partner Organizations (artlook) → PDC Field Mapping

**Subject:** `C:\Data\Parliament\chicago_partner_org_metadata.xlsx` — the `columns` tab (schema inventory) and the `relationships` tab (join contract)
**Supporting context:** the companion `chicago_partner_org_metadata.md` in the same folder, which adds entity groupings, the allocation matrix, and `state`-field semantics that the spreadsheet alone does not carry

### 2.1 The two schemas are structurally identical

A column-level diff confirms `chicago.*` is a **strict subset** of `public.*`: every shared table has identical columns, and `public` additionally carries six lookup/aux tables (`demographics`, `engagement_types`, `integration_types`, `schedule_types`, `sign_ins`, `space_types`).
| All programs for an org, rendered | `proposal_related_programs` | ⭐ **High** |
| `programs.description` | `proposal_activities`, `proposal_summary` / `proposal_abstract` | ✅ High |
| `programs.discipline_id` → `chicago.disciplines.name` | `proposal_program_area`, `proposal_focus` | ✅ High |
| `programs.school_year_id` | `proposal_duration` / → `goodAsOf` | ⚠️ Low — a year label, not a duration |

### EOBMF join — ~15 PDC fields, already ingested

They join the full IRS Business Master File but surface only two derived booleans from it.
- **Both datasets map well, but asymmetrically.** The **Profiles** dataset (163,004 foundations, 63 fields) maps cleanly onto **~40 existing PDC base fields** — it is essentially a pre-parsed 990-PF organization record and PDC already has targets for nearly all of it. The **Grantees** dataset (4,882,146 grant records, 21 fields) maps to only **~12** — not because the data is poor, but because **PDC has no grant/award object**, and because of the identity problem below.
- **⚠️ The single biggest blocker is that grantee records carry no EIN.** IRS Form 990-PF Part XV does not require the filer to supply a recipient EIN (unlike Form 990 Schedule I). Grantee identity is a free-text name the funder typed. Without an EIN there is no reliable join to a PDC changemaker, and 4.9M name strings cannot be safely resolved into organization records.
- **Grantmakers.io is already solving that**, and it is the highest-value thing to ask them for. Their typings define an `Enhancements` layer with an LLM-derived **`grantee_ein`** carrying a confidence grade, plus `grantee_classification`, `grantee_mission`, and a `privacy_shield` flag for individual recipients. None of it is in the public index today. **If PDC can obtain the enhanced grants collection, the Grantees mapping goes from ~12 fields to a genuine changemaker-plus-funding-history feed.**
- **The largest untapped org-level source is the EOBMF join the pipeline already performs.** `shared/typings/irs/all.ts` shows grantmakers.io ingests the full IRS Exempt Organizations Business Master File (`NTEE_CD`, `SUBSECTION`, `CLASSIFICATION`, `FOUNDATION`, `DEDUCTIBILITY`, `RULING`, `STATUS`, addresses, financial bands) but surfaces only **two** derived booleans from it. PDC has exact base-field targets for **~15 more** of those columns — including `organization_ntee_code`, which PDC wants and grantmakers.io already holds.
| Title: "Three funder reports from one participant record…" | `proposal_project_title` |
| "The need, in the org's words" | `proposal_need`, `proposal_context` |
| "What a volunteer would do" (7 numbered steps) | `proposal_activities`, `proposal_tactics_and_methods` |
| 5–8 weeks, in a between-cohort window | `proposal_duration`, `proposal_start_date` |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

@bickelj-agent bickelj-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all 10 new mapping docs against the live PDC base-field catalog; the field names and types check out. Copilot thread status in the current documents: 0 resolved, 21 unresolved (3 partially addressed, 18 untouched). Three new issues are in-line. — DeepSeek-V4.1-Flash

| `zip` | `organization_postal_code` | string | ✅ High (`NOT NULL`) |
| `latitude` | `organization_latitude_epsg_4326` | string | ✅ High |
| `longitude` | `organization_longitude_epsg_4326` | string | ✅ High |
| `community_area` | `organization_neighborhood` | string | ⭐ **High** | Chicago's 77 community areas — a genuinely good fit for a field PDC rarely gets filled |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This row has a fifth cell, but the §3.3 table header has four columns. The table renders wrong and this file fails CI: npx prettier --check docs/mappings warns on this file, and the prettier job runs npm run lint:prettier. Give the table a 'Notes' column or drop the trailing cell, then run npm run format:prettier. — DeepSeek-V4.1-Flash


1. **`organization_status` = `"stable"` is the wrong `status`.** The bundle's frontmatter `status:` is an OKF _document lifecycle_ value (`stable` / `draft` — `technology/capability.md` uses `draft`). PDC's `organization_status` is a **public** field describing the organization. Mapping one to the other publishes a document state as an organizational fact. The organization's actual status lives in `verification.md` ("Current status: ELIGIBLE") and belongs in `review_status`.

2. **`proposal_name` names the wrong organization** — "synthetic-Cumberland Gap Health Cooperative — What I need funding for", on the Black Mountain record. This is an upstream content bug in `what_i_need_funding_for.md`, but the exporter propagates it verbatim. Worth an ingest-time assertion that a proposal title's organization prefix matches the changemaker.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This claim is inverted. In the live what_i_need_funding_for.md, the frontmatter title is synthetic-Black Mountain Workforce Partnership — What I need funding for. So proposal_name holds the correct organization. Only the body carries the clinic text. Document it as a body/title mismatch, not a wrong-org proposal_name. The same error is at line 169 and in the summary (line 84). — DeepSeek-V4.1-Flash

- **Yes — substantially more.** The feed maps **14 of 282** base fields. Another **~55 existing base fields** can be populated from the bundle as it stands today, with no new PDC fields required. That is roughly a **4× increase in coverage** from the same source files.
- **The root cause is mechanical, not semantic.** The exporter reads only YAML **frontmatter** (`title`, `description`, and four `x-civic` keys). It never reads the markdown **body** — and the body is where every number, the mission statement, the address, the financials, the program list, and the verification determination live. Fixing the extraction layer, not the mapping vocabulary, is where the leverage is.
- **Four highest-value additions, all trivially available:** `organization_website` (already sits in frontmatter as `resource:` and is simply dropped), `organization_mission_statement`, the address quintet (`organization_street_address_1` / `city` / `state_province` / `postal_code` / `county`), and the financial trio (`organization_total_revenue`, `organization_total_assets`, `organization_paid_staff`).
- **Three genuine data-quality bugs** in the current 14 mappings should be fixed regardless of scope: `organization_status` carries a _document_ lifecycle value, `proposal_name` names the **wrong organization**, and three narrative fields carry the file's meta-disclaimer instead of its content. See §5.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

three narrative fields contradicts §5 item 3 and the summary gotcha, which both say four values. This bullet also counts three bugs, while §5 lists seven and §9 says six. Use one count in all three places. — DeepSeek-V4.1-Flash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants