crytpto_data_pull.ipynb

{
  "cells": [
    {
      "metadata": {
        "_uuid": "7bee07eb8417bcb056427d60567aefa875b9f9f2"
      },
      "cell_type": "markdown",
      "source": "forked from Cryptocurrency Data Pull by SRK (source: https://www.kaggle.com/sudalairajkumar/cryptocurrency-data-pull) \n Since it is getting harder to update this every week and also some people might need it more often than a week, I have put together everything into python code.\n\nPlease use the below code to pull the data from the websites mentioned in the data overview section.\n\nAll the data credits go to the corresponding owners. Please make sure you mention their names if you use it somewhere.\n\n**Code to get data from coin market cap:**\n\nPlease uncomment the \"get_data\" function in the last line run it in local"
    },
    {
      "metadata": {
        "_uuid": "c5576ea8237ca208fea192217e38be50fd40ba27",
        "_cell_guid": "749d23cd-92bc-4a4c-af2c-a3a4a47b803a",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "# -*- coding: utf-8 -*-\nimport re\nimport sys\nimport csv\nimport time\nimport random\nimport requests\nfrom datetime import date\nfrom bs4 import BeautifulSoup\n\nend_date = str(date.today()).replace(\"-\",\"\")\nbase_url = \"https://coinmarketcap.com/currencies/{0}/historical-data/?start=20130428&end=\"+end_date\n\ncurrency_name_list = [\"bitcoin\", \"ethereum\", \"ripple\", \"bitcoin-cash\", \"nem\", \"litecoin\", \"dash\", \"ethereum-classic\", \"iota\", \"neo\", \"stratis\", \"monero\", \"waves\", \"bitconnect\", \"omisego\", \"qtum\", \"numeraire\"]\n\n\ndef get_data(currency_name):\n    print(\"Currency : \", currency_name)\n    url = base_url.format(currency_name)\n    html_response = requests.get(url).text.encode('utf-8')\n    soup = BeautifulSoup(html_response, 'html.parser')\n    table = soup.find_all('table')[0]\n    elements = table.find_all(\"tr\")\n    with open(\"./{0}_price.csv\".format(currency_name.replace(\"-\",\"_\")),\"w\") as ofile:\n        writer = csv.writer(ofile)\n        for element in elements:\n            writer.writerow( element.get_text().strip().split(\"\\n\") )\n    time.sleep(1)\n\nif __name__ == \"__main__\":\n    for currency_name in currency_name_list:\n        #get_data(currency_name)\n        pass",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "8272383273819c5f68329b9fecca350f3ca82186"
      },
      "cell_type": "markdown",
      "source": "**Code to get bitcoin dataset:**\n\nCode to get the features from blockchain info site. Please uncomment the function call in local to run."
    },
    {
      "metadata": {
        "_uuid": "eef939c61c9c7e0ea7ad8db2d7c4d67fa1f6ebd2",
        "trusted": false,
        "collapsed": true
      },
      "cell_type": "code",
      "source": "import time\nimport requests\nimport pandas as pd\n\nurls = [\n'https://blockchain.info/charts/market-price',\n'https://blockchain.info/charts/total-bitcoins',\n'https://blockchain.info/charts/market-cap',\n'https://blockchain.info/charts/trade-volume',\n'https://blockchain.info/charts/blocks-size',\n'https://blockchain.info/charts/avg-block-size',\n'https://blockchain.info/charts/n-orphaned-blocks',\n'https://blockchain.info/charts/n-transactions-per-block',\n'https://blockchain.info/charts/median-confirmation-time',\n'https://blockchain.info/charts/hash-rate',\n'https://blockchain.info/charts/difficulty',\n'https://blockchain.info/charts/miners-revenue',\n'https://blockchain.info/charts/transaction-fees',\n'https://blockchain.info/charts/cost-per-transaction-percent',\n'https://blockchain.info/charts/cost-per-transaction',\n'https://blockchain.info/charts/n-unique-addresses',\n'https://blockchain.info/charts/n-transactions',\n'https://blockchain.info/charts/n-transactions-total',\n'https://blockchain.info/charts/n-transactions-excluding-popular',\n'https://blockchain.info/charts/n-transactions-excluding-chains-longer-than-100',\n'https://blockchain.info/charts/output-volume',\n'https://blockchain.info/charts/estimated-transaction-volume',\n'https://blockchain.info/charts/estimated-transaction-volume-usd'\n]\n\nsuffix_to_add = '?timespan=8years&format=csv'\n\ndef get_btc_data():\n    counter = 0\n    for url in urls:\n        header = ['Date', \"btc_\" + url.split(\"/\")[-1].replace(\"-\",\"_\")]\n        print(header[-1])\n        temp_df = pd.read_csv(url+suffix_to_add, header=None, names=header)\n        if counter == 0:\n            df = temp_df.copy()\n        else:\n            df = pd.merge(df, temp_df, on=\"Date\", how=\"left\")\n        print(temp_df.shape, df.shape)\n        counter += 1\n        time.sleep(1)\n    df.to_csv(\"../input_v9/bitcoin_dataset.csv\", index=False)\n    \n#get_btc_data()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "fe9aa45c6948aefa0ced1929483dc707dc99ef6f"
      },
      "cell_type": "markdown",
      "source": "**Code to get Ethereum dataset from EtherScan:**\n\nPlease find below the code to get ethereum related info from etherscan.io. Please uncomment the last line in local and run."
    },
    {
      "metadata": {
        "collapsed": true,
        "_uuid": "3e81b84f2cd9f6132afbcd2782a89ebff2f85a81",
        "trusted": false
      },
      "cell_type": "code",
      "source": "import time\nimport requests\nimport pandas as pd\n\nurls = [\n'https://etherscan.io/chart/etherprice',\n'https://etherscan.io/chart/tx',\n'https://etherscan.io/chart/address',\n'https://etherscan.io/chart/marketcap',\n'https://etherscan.io/chart/hashrate',\n'https://etherscan.io/chart/difficulty',\n'https://etherscan.io/chart/blocks',\n'https://etherscan.io/chart/uncles',\n'https://etherscan.io/chart/blocksize',\n'https://etherscan.io/chart/blocktime',\n'https://etherscan.io/chart/gasprice',\n'https://etherscan.io/chart/gaslimit',\n'https://etherscan.io/chart/gasused',\n'https://etherscan.io/chart/ethersupply',\n'https://etherscan.io/chart/ens-register'\n]\n\nsuffix_to_add = '?output=csv'\n\ndef get_ether_data():\n    counter = 0\n    for url in urls:\n        header = ['Date', 'TimeStamp', \"eth_\" + url.split(\"/\")[-1].replace(\"-\",\"_\")]\n        print(header[-1])\n        \n        with open(\"temp.csv\", \"w\") as ofile:\n            response = requests.get(url+suffix_to_add).text.encode('utf-8')\n            ofile.write(response)\n        temp_df = pd.read_csv(\"temp.csv\")\n        \n        col_names = temp_df.columns.tolist()\n        if col_names[-1] == \"Value\":\n            col_names = col_names[:2] + [header[-1]]\n            temp_df.columns = col_names\n        else:\n            temp_df = temp_df[[\"Date(UTC)\",\"UnixTimeStamp\", \"Supply\", \"MarketCap\"]]\n            temp_df.columns = [\"Date(UTC)\",\"UnixTimeStamp\", \"eth_supply\", \"eth_marketcap\"]\n            \n        if counter == 0:\n            df = temp_df.copy()\n        else:\n            df = pd.merge(df, temp_df, on=[\"Date(UTC)\",\"UnixTimeStamp\"], how=\"left\")\n        print(temp_df.shape, df.shape)\n        counter += 1\n        time.sleep(1) \n    df.to_csv(\"../input_v9/ethereum_dataset.csv\", index=False)\n    \n#get_ether_data()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "c2b01c6037ad3469d8f0477fd97945d56f805d2e"
      },
      "cell_type": "markdown",
      "source": "Thank you.!"
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}