Skip to content

Latest commit

 

History

History
108 lines (81 loc) · 2.38 KB

README.md

File metadata and controls

108 lines (81 loc) · 2.38 KB

NFT Metadata Scraper

This application scrapes NFT metadata from IPFS using a CSV list of IPFS CIDs and stores the results in a PostgreSQL database. It also hosts an API that allows users to retrieve all stored metadata or a specific row based on a CID.

Features

  • Read a list of IPFS CIDs from a CSV file
  • Fetch metadata for each CID from IPFS
  • Store the name and image fields in a PostgreSQL database
  • Provide an API to retrieve all data or a specific row based on a CID

Prerequisites

  • Go 1.19 or higher
  • PostgreSQL
  • Docker (for running PostgreSQL in a container, if desired)

Setup

Step 1: Clone the Repository

git clone https://github.com/shawnwollenberg/ipfs-metadata.git

Step 2: Set Up PostgreSQL

You can either set up PostgreSQL locally or use Docker to run it in a container.

Using Docker:

docker run --name postgres -e POSTGRES_USER=youruser -e POSTGRES_PASSWORD=yourpassword -e POSTGRES_DB=yourdb -p 5432:5432 -d postgres

Step 3: Configure Environment Variables

Create a .env file in the root directory of the project with the following content:

env

POSTGRES_USER=youruser
POSTGRES_PASSWORD=yourpassword
POSTGRES_DB=yourdb
POSTGRES_HOST=localhost
POSTGRES_PORT=5432

Step 4: Install Dependencies

go mod tidy

Step 5: Prepare the CSV File

A CSV file has been saved in the data directory, but if you would like to add additional CIDs feel free to adjust. Each row should contain one CID.

Step 6: Run the Application

go run .

This will:

  • Read the CSV file.
  • Fetch metadata for each CID from IPFS.
  • Store the name and image fields in the PostgreSQL database.
  • Start the API server.

API Endpoints

Get All Metadata

Request:
GET /metadata
Response:
[
  {
    "cid": "Qm...",
    "name": "Example Name",
    "image": "Example Image URL"
  },
  ...
]

Get Metadata by CID

Request:

GET /metadata/:cid

Response:
{
  "cid": "Qm...",
  "name": "Example Name",
  "image": "Example Image URL"
}

Acknowledgements

  • Gin Gonic for the web framework.
  • sqlx for SQL database interactions.
  • godotenv for loading environment variables from a .env file.

Contact

For any questions or suggestions, please open an issue or contact the repository owner.