-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
16 changed files
with
466 additions
and
103 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| HTTP_PORT=3282 | ||
| DATABASE_HOST=localhost | ||
| DATABASE_PORT=27017 | ||
| DATABASE_PORT=3306 | ||
| DATABASE_USER=root | ||
| DATABASE_PASSWORD=root | ||
| DATABASE_PASSWORD=passw00d | ||
| DATABASE_NAME=blobs | ||
| METRICS_HTTP_PORT=7471 | ||
| METRICS_HTTP_PORT=7471 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,55 +1,93 @@ | ||
| # blob-storage | ||
|
|
||
| Repo for BLOB storage (archive and serve data) | ||
| Repository for BLOB storage (archive and serve data) | ||
|
|
||
| ## how to run ? | ||
| ## Prerequisites | ||
|
|
||
| Prerequisite is to have docker engine up and running. | ||
| - Docker engine up and running. | ||
| - Go installed on your system. | ||
|
|
||
| 1. Start the mongoDB | ||
| ## Configuration and Setup | ||
|
|
||
| ```bash | ||
| cd local_docker && docker-compose up -d | ||
| ``` | ||
| ### Setting up MySQL | ||
|
|
||
| 2. Start the `indexer` | ||
| 1. **Start MySQL**: | ||
|
|
||
| ```bash | ||
| ENV_FILE=.default.indexer.env run cmd/main.go indexer | ||
| ``` | ||
| Navigate to the `docker-compose` directory and start the MySQL service: | ||
|
|
||
| By default the above command starts the app from the latest block height. If we want to specify a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. | ||
| ```bash | ||
| cd ./docker-compose | ||
| docker-compose up -d | ||
| ``` | ||
|
|
||
| 2. Start the `server`. | ||
| This command starts your MySQL instance as defined in your `docker-compose.yml` file. | ||
|
|
||
| ```bash | ||
| ENV_FILE=.default.server.env run cmd/main.go server | ||
| ``` | ||
| 2. **Migrate Database Schema**: | ||
|
|
||
| ## how to test / use ? | ||
| Navigate to the `migrations` directory to apply database migrations: | ||
|
|
||
| When the `DB`, `blob-catcher` and `server` is running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, tho it is not written into the DB). Use that `blobHash` (including the 0x) in | ||
| ```bash | ||
| cd ./migrations | ||
| goose mysql "root:passw00d@tcp(localhost:3306)/blobs" status | ||
| goose mysql "root:passw00d@tcp(localhost:3306)/blobs" up | ||
| ``` | ||
|
|
||
| 1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array): | ||
| These commands apply migrations to the `blobs` database. | ||
|
|
||
| ```bash | ||
| curl -X GET "http://localhost:27001/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" | ||
| ``` | ||
| ### Environment Configuration | ||
|
|
||
| The result will be something like this: | ||
| Ensure your `.default.indexer.env` and `.default.server.env` files are configured with the correct database credentials, host, and any other necessary environment variables. | ||
|
|
||
| ```bash | ||
| {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} | ||
| ``` | ||
| ## Running the Application | ||
|
|
||
| 2. Or to backtest, use the simple python script below, after overwriting the `blob_hash` variable: | ||
| 1. **Start the Indexer**: | ||
|
|
||
| ```bash | ||
| python3 python_query.py | ||
| ``` | ||
| With the environment file configured, start the indexer: | ||
|
|
||
| ## todos | ||
| ```bash | ||
| ENV_FILE=.default.indexer.env go run cmd/main.go indexer | ||
| ``` | ||
|
|
||
| This starts the app from the latest block height by default. Adjust the `STARTING_BLOCK_ID` in the environment file if needed. | ||
|
|
||
| 2. **Start the Server**: | ||
|
|
||
| Similarly, start the server: | ||
|
|
||
| ```bash | ||
| ENV_FILE=.default.server.env go run cmd/main.go server | ||
| ``` | ||
|
|
||
| ## Testing and Usage | ||
|
|
||
| When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, though it is not written into the DB). Use that `blobHash` (including the 0x) in | ||
|
|
||
| 1. **Querying Blob Data via HTTP Request**: | ||
|
|
||
| To retrieve blob data, you can execute a `curl` command. This allows for querying multiple `blobHashes` simultaneously, separated by commas. A single request can yield an array of results: | ||
|
|
||
| ```bash | ||
| curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" | ||
| ``` | ||
|
|
||
| **Expected Output**: | ||
|
|
||
| ```bash | ||
| {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} | ||
| ``` | ||
|
|
||
| 2. **Backtesting with a Python Script**: | ||
|
|
||
| This script facilitates querying the database directly based on a specified `blob_hash`. Modify the `blob_hash` variable in the script to match the hash you wish to query. | ||
|
|
||
| To run the script: | ||
|
|
||
| ```bash | ||
| python3 python_query.py | ||
| ``` | ||
|
|
||
| ## Todos | ||
|
|
||
| What is still missing is: | ||
|
|
||
| - small refinements and DevOps (prod-grade DB with creditentials, proper containerization) | ||
| - small refinements and DevOps (prod-grade DB with credentials, proper containerization) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| version: "3.8" | ||
| services: | ||
| db: | ||
| image: mysql:8.0 | ||
| cap_add: | ||
| - SYS_NICE | ||
| restart: always | ||
| environment: | ||
| - MYSQL_DATABASE=blobs | ||
| - MYSQL_ROOT_PASSWORD=passw00d | ||
| ports: | ||
| - "3306:3306" | ||
| volumes: | ||
| - ~/.docker-conf/mysql/data/:/var/lib/mysql |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| // Package docs Code generated by swaggo/swag. DO NOT EDIT | ||
| package docs | ||
|
|
||
| import "github.com/swaggo/swag" | ||
|
|
||
| const docTemplate = `{ | ||
| "schemes": {{ marshal .Schemes }}, | ||
| "swagger": "2.0", | ||
| "info": { | ||
| "description": "{{escape .Description}}", | ||
| "title": "{{.Title}}", | ||
| "termsOfService": "http://swagger.io/terms/", | ||
| "contact": { | ||
| "name": "API Support", | ||
| "url": "https://community.taiko.xyz/", | ||
| "email": "[email protected]" | ||
| }, | ||
| "license": { | ||
| "name": "MIT" | ||
| }, | ||
| "version": "{{.Version}}" | ||
| }, | ||
| "host": "{{.Host}}", | ||
| "basePath": "{{.BasePath}}", | ||
| "paths": { | ||
| "/getBlob": { | ||
| "get": { | ||
| "consumes": [ | ||
| "application/json" | ||
| ], | ||
| "produces": [ | ||
| "application/json" | ||
| ], | ||
| "summary": "Get blob(s) and KZG commitment(s)", | ||
| "operationId": "get-blob", | ||
| "parameters": [ | ||
| { | ||
| "type": "string", | ||
| "description": "blobHash to query", | ||
| "name": "blobHash", | ||
| "in": "query", | ||
| "required": true | ||
| } | ||
| ], | ||
| "responses": { | ||
| "200": { | ||
| "description": "OK", | ||
| "schema": { | ||
| "$ref": "#/definitions/http.getBlobResponse" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "definitions": { | ||
| "http.blobData": { | ||
| "type": "object", | ||
| "properties": { | ||
| "blob_hash": { | ||
| "type": "string" | ||
| }, | ||
| "kzg_commitment": { | ||
| "type": "string" | ||
| } | ||
| } | ||
| }, | ||
| "http.getBlobResponse": { | ||
| "type": "object", | ||
| "properties": { | ||
| "data": { | ||
| "type": "array", | ||
| "items": { | ||
| "$ref": "#/definitions/http.blobData" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }` | ||
|
|
||
| // SwaggerInfo holds exported Swagger Info so clients can modify it | ||
| var SwaggerInfo = &swag.Spec{ | ||
| Version: "1.0", | ||
| Host: "blobs.internal.taiko.xyz", | ||
| BasePath: "", | ||
| Schemes: []string{}, | ||
| Title: "Taiko Blobstorage API", | ||
| Description: "", | ||
| InfoInstanceName: "swagger", | ||
| SwaggerTemplate: docTemplate, | ||
| LeftDelim: "{{", | ||
| RightDelim: "}}", | ||
| } | ||
|
|
||
| func init() { | ||
| swag.Register(SwaggerInfo.InstanceName(), SwaggerInfo) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| <html> | ||
| <head> | ||
| <!-- Load the latest Swagger UI code and style from npm using unpkg.com --> | ||
| <script src="https://unpkg.com/swagger-ui-dist@3/swagger-ui-bundle.js"></script> | ||
| <link rel="stylesheet" type="text/css" href="https://unpkg.com/swagger-ui-dist@3/swagger-ui.css"/> | ||
| <title>Blobstorage API</title> | ||
| </head> | ||
| <body> | ||
| <div id="swagger-ui"></div> <!-- Div to hold the UI component --> | ||
| <script> | ||
| window.onload = function () { | ||
| // Begin Swagger UI call region | ||
| const ui = SwaggerUIBundle({ | ||
| url: "/swagger.json", //Location of Open API spec in the repo | ||
| dom_id: '#swagger-ui', | ||
| deepLinking: true, | ||
| presets: [ | ||
| SwaggerUIBundle.presets.apis, | ||
| SwaggerUIBundle.SwaggerUIStandalonePreset | ||
| ], | ||
| plugins: [ | ||
| SwaggerUIBundle.plugins.DownloadUrl | ||
| ], | ||
| }) | ||
| window.ui = ui | ||
| } | ||
| </script> | ||
| </body> | ||
| </html> |
Oops, something went wrong.