Research
Indian Parliamentary Data
A restartable pipeline that turns parliamentary documents and campaign audio into structured research data.
- Setting
- CU Boulder Studio Lab · Princeton University
- Status
- Princeton appointment completed · May 2026
Overview
The problem
Political archives span dynamic pages, long recordings, PDFs, and inconsistent ministry names. Researchers need traceable outputs without restarting hours of collection after an interruption.
My contribution
I designed comparative speech and translation evaluations at Studio Lab. In a later Princeton appointment, I built archive scraping, orchestration, PDF extraction, normalization, and resumable processing.
Interactive example · synthetic data
From source text to a recoverable record
An invented document and simplified checkpoint illustrate the pipeline. These are not research records or measured processing results.
Step 1 of 4
Capture
Retain the source reference alongside extracted text so a transformation can be traced back to its input.
- Source document
- statement-demo-003.pdf · page 2
- Extracted ministry
- Min. of Example Affairs
- Extracted statement
- The committee reviewed the proposed programme.
Step 2 of 4
Normalize
Map a candidate name into a consistent representation while keeping the original text available for review.
- Raw ministry
- Min. of Example Affairs
- Candidate normalized name
- Ministry of Example Affairs
- Review status
- Needs verification
- Provenance
- statement-demo-003.pdf · page 2
Step 3 of 4
Interrupt
An interrupted job should not discard already completed records or advance its checkpoint past unfinished work.
- Completed records
- demo-001, demo-002
- Pending record
- demo-003
- Retained artifacts
- Source text and completed normalized records
Step 4 of 4
Resume
Resume from the retained boundary. In this simplified example, the first two records are reused and the third is processed.
- Restart point
- demo-003
- Work retained
- demo-001 and demo-002
- Recovery state
- Waiting to resume
Illustrated recovery complete: demo-001 and demo-002 were retained; demo-003 was processed. Its candidate ministry name still needs review.
Measured and implemented
Results, with their scope.
- 40+ GB of audio
- 2014 and 2019 campaign recordings processed after tool evaluation.
- Multi-decade archives
- Lok Sabha and Rajya Sabha records collected for research.
- Thousands of PDFs
- Parliamentary statements parsed and mapped to standardized ministry names.
Architecture
How the parts work together
Select a step for the decision behind it.
Tool evaluation
Speech and translation services are compared against the needs of the research workflow before they become pipeline dependencies.
Design details for Tool evaluation
At Studio Lab, I set up WER-oriented comparisons across Deepgram, Google Speech-to-Text, Amazon Transcribe, and Azure Speech, plus translation comparisons across Google, AWS, and Azure services.
Archive scraping
Dynamic parliamentary pages and documents become a recoverable source collection.
Design details for Archive scraping
At Princeton, I built the Selenium scraping logic, handled pagination, and persisted progress in SQLite so long collection runs could resume.
End-to-end orchestration
Audio, page metadata, and documents move through one inspectable research workflow.
Design details for End-to-end orchestration
The workflow coordinates source capture, speech and translation processing, PDF text extraction, and intermediate artifacts instead of relying on disconnected one-off scripts.
Data structuring
Inconsistent records become formats that researchers can inspect and analyze.
Design details for Data structuring
PyMuPDF extraction, metadata normalization, and fuzzy ministry-name mapping produce structured outputs for downstream analysis.
Recovery and review
A failed stage can restart without discarding completed work or hiding transformation decisions.
Design details for Recovery and review
Saved state, intermediate outputs, and reviewable mappings keep processing recoverable and normalization decisions auditable.
Studio Lab: evaluation phase
At CU Boulder’s Studio Lab, my primary work was comparative research. I designed word-error-rate evaluations across Deepgram, Google Speech-to-Text, Amazon Transcribe, and Azure Speech, then set up parallel translation comparisons across Google Cloud Translation, Amazon Translate, and Azure Translator. I also researched configuration and language-adaptation options for political names and domain-specific vocabulary.
The goal was not to promote one vendor. It was to understand how standard tools behaved on the project’s material and identify which components fit the workflow.
Princeton: engineering phase
Under the later Princeton appointment, I built the scraping logic and orchestrated the workflow end to end. That included dynamic Parliament Digital Library pagination, resumable SQLite state, speech and translation processing for more than 40 GB of 2014 and 2019 campaign audio, PDF text extraction with PyMuPDF, and structured outputs for further analysis.
I parsed thousands of parliamentary statements and used fuzzy matching to map inconsistent ministry names into a standardized research representation. Checkpoints and intermediate artifacts made long-running jobs restartable and kept transformations open to review.
Engineering decisions
- Evaluate interchangeable services before embedding one vendor into the research workflow.
- Persist collection state so interrupted sessions can resume near the failure rather than restarting.
- Keep raw source material, intermediate text, and normalized outputs distinct so transformations can be inspected.
- Treat source text and fuzzy matching as assistive normalization inputs that still need review, not unquestioned ground truth.
- Separate the campaign-audio corpus from the parliamentary-document corpus so scale and date claims are not merged.
Research context
This project spanned two separate paid appointments: Studio Lab at CU Boulder (January 21–April 13, 2024) and Princeton (January 15, 2025–May 2026), with continued research collaboration between them. Experience lists the roles and their distinct contributions.
Technologies
Python · Selenium · SQLite · AWS · Google Cloud · Azure · Deepgram · PyMuPDF
Get in touch
Discuss this project.
Questions about Indian Parliamentary Data, the tradeoffs, or my contribution?