01 / Making data useful at scale

Raw data in.
Decision-ready
assets out.

Currently working on/ 001● Load
5.2M+ records
PostgreSQL dimensional warehouse
DATA ENGINEER / MUMBAI, INDIASCROLL TO EXPLORE

Dharmesh Kashyap

I build pipelines, warehouses, and validation systems that make difficult data dependable.

000/ Mindset

Government surveys, scheme PDFs, boundary files, Excel uploads. The inputs are never clean. I build the systems that make them trustworthy: pipelines that run on their own, schemas that stay consistent round after round, and numbers a decision maker can lean on.

2+ years in productionSeamless Digital Automations, MumbaiSoftware Developer (Data Engineer), since Feb 2025
001 / Evidence in production

Numbers
that shipped.

Not vanity metrics. A cross-section of real delivery- from national geospatial data and survey harmonization to reliable finance infrastructure.

01 / Spatial engineering↗ NABARD
670K+
Village boundaries processed and delivered as GeoJSON
Pan-India geospatial reporting
02 / Data infrastructure↗ ICICI PRU
5.2M+
Records in a staging-to-dimension/fact warehouse
PostgreSQL / SQL optimization
03 / Document extraction↗ NABARD
2,500+
Government scheme PDFs handled by validated extraction pipelines
Python / structured extraction
04 / Survey harmonization↗ NSO / MoSPI
298
Harmonized variables in a three-round HCES DDI dictionary
Repeatable definitions / data quality
05 / Survey coverage↗ NATIONAL DATA
5 × 3
National surveys processed across three rounds
PySpark / Hadoop / Polars / DuckDB
06 / Recurring automation↗ MARKETS
60+
Mutual fund instruments consolidated into a single schema
Scheduled extraction / reconciliation
Selected production outcomes / figures sourced from latest resumeScroll to see the systems behind them ↓
002/ The pipeline. Scroll to walk it.

Extract

The inputs are never clean.

Boundary files for 670K+ Pan India villages. 2,500+ government scheme PDFs. 60+ mutual fund instruments, consolidated on a recurring pipeline into one schema. Extraction ships with validation checks, so problems surface at the door.

GeoJSON. PDF. Python. Validation.

Transform

Five surveys. One schema.

HCES, ASUSE, AIDIS, SAS and the Agriculture Census, across three rounds of NSO and MoSPI microdata. PySpark and Hadoop do the heavy lifting, Polars and DuckDB shape and validate, and BI tools read the long format output directly. A DDI dictionary of 298 harmonized variables keeps definitions steady from round to round.

PySpark. Hadoop. Polars. DuckDB. Parquet.

Load

Staging in. Facts out.

A PostgreSQL warehouse over 5.2M+ records for ICICI Prudential. Staging tables are promoted into dimension and fact tables through a multi step pipeline, with query optimization for analytics. The Data Processor app puts a Flask front door on it.

PostgreSQL. Flask. SQL. Dimensional modeling.

Serve

Numbers people can use.

KPI dashboards with drill downs. AIV Hub deployed on AWS hosted servers, from setup and configuration to monitoring. When a cross subnet routing failure appeared, the fix came with written network and firewall requirements.

Tableau. AIV Hub. AWS.

003 / The transformation / Scroll to follow

Wide in.
Long out.

Survey data often arrives as one column per variable. I reshape it into a consistent long-format dataset that BI tools can consume directly.

↳ONE INPUT.
ONE REUSABLE SCHEMA.
NO MANUAL REFORMAT.

Follow the two states as you scroll
Illustrative values, not real data.

01 / SOURCE DATAWIDE FORMAT
idv1v2v3
r1a1b1c1
r2a2b2c2
r3a3b3c3
r4a4b4c4
FOUR RECORDS / THREE VARIABLE COLUMNS
02 / STANDARDIZED OUTPUTLONG FORMAT ✓
idvariablevalue
r1v1a1
r1v2b1
r1v3c1
r2v1a2
r2v2b2
r2v3c2
r3v1a3
r3v2b3
r3v3c3
r4v1a4
r4v2b4
r4v3c4
ONE CONSISTENT SCHEMA / READY FOR ANALYTICS
004/ Case files

Data Processor

Flask. PostgreSQL. ICICI Prudential. 2025 to 2026.

Finance teams were uploading Excel files manually. This app takes Admin Dump and Location Vacation files into staging tables, validates them, and promotes them into the dimension and fact warehouse.

It grew a scheduler view, refresh progress, conflict reporting and a notification tray, and went through an audit that fixed duplicate promotion and staging filter bugs.

  1. Excel upload
  2. Staging
  3. Validate
  4. Dimension and fact

News Intelligence Platform

Built for NABARD. 2026.

A multi source scraper extracts, cleans and structures articles into a database that feeds a live dashboard.

AI enrichment adds classification, summarization and search, alongside an interactive commodities ticker.

  1. Sources
  2. Scrape and clean
  3. AI enrichment
  4. Live dashboard
005 / Technical capabilities

The stack,
in context.

Tools aren't the story. What they enable is. Scroll through four layers of the systems I build, from raw inputs to useful decisions.

CAPABILITY / 01 ↗
PROCESS / 01

From unruly input to consistent data.

Distributed extraction and transformation built for changing source formats, survey rounds, and high-volume workloads.

IN PRACTICE →Five NSO/MoSPI surveys, harmonized across three rounds; HCES delivered as a testable Python ETL package.
PythonPySparkHadoopPolarsDuckDBParquetData validation
CAPABILITY / 02 ↗
STRUCTURE / 02

Warehouses designed to stay trustworthy.

From raw staging and business rules to dimensional models, predictable joins, and performant analytical queries.

IN PRACTICE →ICICI Prudential warehouse with 5.2M+ records and staged promotion into dimension and fact tables.
PostgreSQLMySQLSQLDimensional modelingQuery optimizationETL design
CAPABILITY / 03 ↗
SYSTEMS / 03

Put reliable pipelines into operation.

Backend services, scheduling visibility, conflict reporting, and cloud deployments that turn a one-off script into a repeatable workflow.

IN PRACTICE →Flask Data Processor for file-to-database promotion and AIV Hub deployment on AWS-hosted infrastructure.
FlaskAWSPythonAutomationMonitoringValidation
CAPABILITY / 04 ↗
DECISIONS / 04

Bring trusted numbers to decision-makers.

Dashboards and KPI frameworks that help people move from high-level performance to meaningful drill-downs.

IN PRACTICE →KPI reporting and analytics delivery for enterprise finance and government data initiatives.
TableauAIV HubKPI reportingDashboard developmentSQL
006 / Foundations

LEARNING, APPLIED

Built on
solid
ground.

Formal training in computing, followed by hands-on work across data systems, cloud infrastructure and analytics.

2023 — 202502 / MASTER'S

Master of Computer Applications (MCA)

Manipal Institute of Technology · Manipal Academy of Higher Education

DATA ANALYTICSDATABASE MANAGEMENTCLOUD COMPUTING
2019 — 202201 / BACHELOR'S

Bachelor of Science in Information Technology (B.Sc. IT)

· University of Mumbai

CONTINUED LEARNING / CERTIFICATIONS

01 Modern Application Development with Python on AWS

02 Applied Data Science with Python

Messy data? Let's talk.

dharmeshkashyap46@gmail.com

Open to data engineering roles, contract pipelines, and consulting on analytics infrastructure.

Mumbai, India. +91 89282 10971.