Skip to content

Latest commit

 

History

History
113 lines (83 loc) · 2.64 KB

README.md

File metadata and controls

113 lines (83 loc) · 2.64 KB

PostgreSQL Bulk Data Generator

A high-performance data generation and bulk import tool built with Bun and PostgreSQL. This tool demonstrates efficient bulk data operations by generating millions of realistic test records and importing them using PostgreSQL's COPY command.

Features

  • 🚀 Fast data generation using Bun's performance capabilities
  • 📊 Generates realistic test data using Faker.js
  • 📝 Efficient CSV file handling
  • 🔄 Uses PostgreSQL's COPY command for optimal bulk imports
  • 📈 Progress logging and performance metrics
  • 🐳 Docker-ready PostgreSQL setup

Prerequisites

  • Bun (v1.1.43 or later)
  • Docker (for PostgreSQL container)

Installation

  1. Clone the repository:
git clone https://github.com/vdelacou/postgresql-bulk-data-generator.git
cd postgresql-bulk-data-generator
  1. Install dependencies:
bun install
  1. Start the PostgreSQL container:
docker run -d \
  --name postgres_bulk_test \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=mydb \
  -p 5432:5432 \
  postgres:15-alpine

Usage

  1. Run the data generation and import script:
bun run src/index.ts

The script will:

  • Generate 1 million records with realistic user data
  • Create a CSV file
  • Import the data into PostgreSQL
  • Clean up temporary files automatically

Configuration

You can modify the following settings in src/index.ts:

  • itemsCount: Number of records to generate
  • tableName: Target table name
  • columns: Database table schema
  • db: Database connection settings

Useful Commands

Check record count:

docker exec -it postgres_bulk_test psql -U postgres -d mydb -c "SELECT COUNT(*) FROM consumers;"

Clear the table:

docker exec -it postgres_bulk_test psql -U postgres -d mydb -c "DELETE FROM consumers;"

Docker Container Management

Start the container:

docker start postgres_bulk_test

Stop the container:

docker stop postgres_bulk_test

Check container status:

docker ps

Performance

On a typical development machine, this tool can:

  • Generate 1 million records
  • Import data using PostgreSQL's COPY command for optimal performance
  • Complete the entire process in few seconds

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

  • Bun for the blazing-fast JavaScript runtime
  • Faker.js for realistic data generation
  • node-postgres for PostgreSQL interaction