{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a1ebc14",
   "metadata": {},
   "source": [
    "#Cosmic Ray and Earthquake analysis\n",
    "Last updated 21/9/21 J. Devine\n",
    "\n",
    "##This is what I was planning to do - I didn't get very far.\n",
    "\n",
    "Stage 1: Collect the data\n",
    "datasets - nmdb data (all stations, corrected rates) (done!)\n",
    "USGS Earthquake db (done!)\n",
    "Sunspot data - cross ref with cosmics and earthquakes.\n",
    "Weather data - nearest neighbour for each NMDB location (not done - would be nice in the future)\n",
    "\n",
    "\n",
    "Stage 2: Back-calc the pressure correction - not done!\n",
    "    a - cross ref. air pressure with nearest neighbour\n",
    "    b - compare raw and corrected counts\n",
    "    c - compare delta with pressure data sets\n",
    "    d - compute correction constant\n",
    "    e - compute pressure /cosmic rate correlation\n",
    "\n",
    "Then repeat for sunspot data. sunspot correlation with cosmics (expected -ve and strong)\n",
    "\n",
    "Stage 3: Earthquake data\n",
    "structure 10 year data (2010-2020), inc. 6 hourly averages of seismic energy. \n",
    "plot all earthquake locations (lat/long) to see fault lines\n",
    "Find the nearest nmdb sensor for each (actually find nearest earthquakes for each nmdb..).\n",
    "Compute for each earthquake the distances to NMDB sites (triangulation)\n",
    "\n",
    "\n",
    "Stage 4:\n",
    "Windows. \n",
    "Window variables - averages from 1m to 4 weeks (compute for cosmic and seismic from raw)\n",
    "Window variable - W\n",
    "Offset variable - look at all offsets from 1m to 365 days (+ and -)\n",
    "In each case perform a regression across the dataset and select for highest correlations.\n",
    "\n",
    "\n",
    "Stage 5: Random variable\n",
    "Pick a random variable from the weather data and do a null analysis.\n",
    "\n",
    "##What I actually did (so far)\n",
    "Import data from NMDB monitors, for basically all the sites, all date ranges and build arrays (pandas), and rescale per period.\n",
    "Import data from USGS and compute average seismic activity per period\n",
    "Import sunspot data and compute average sunsport data per period\n",
    "\n",
    "Correlation between USGS and sunspot data, with p values, across a range of offsets up to +/-10k days- interesting patterns!\n",
    "\n",
    "##What is left to do to make this useful\n",
    "Wrangle the NMDB data properly to do a similar variable offset calc with USGS data. \n",
    "If the correlations and p values are reasonable at specific offsets then - \n",
    "Compare this analysis with c(P,t0,d,m,𝚫t) = A*B = (Sm(ti+𝚫t)/M(Sm(ti+𝚫t))-1))*(|𝚫ncr(ti,i-1)|/M(|𝚫ncr(ti,i-1)|)-1) \n",
    "Build a PDF for agreement etc.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0beec68a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse, csv, urllib.request, time\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "from datetime import datetime\n",
    "from pandas.tseries.offsets import DateOffset\n",
    "import matplotlib.pyplot as plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "587e52ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import kendalltau, pearsonr, spearmanr\n",
    "\n",
    "def kendall_pval(x,y):\n",
    "    return kendalltau(x,y)[1]\n",
    "    \n",
    "def pearsonr_pval(x,y):\n",
    "    return pearsonr(x,y)[1]\n",
    "    \n",
    "def spearmanr_pval(x,y):\n",
    "    return spearmanr(x,y)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a487bffb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9.5\n"
     ]
    }
   ],
   "source": [
    "from platform import python_version\n",
    "print(python_version())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "666c0823",
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_avg(spl_arraylike):\n",
    "  return 10*np.log10(np.mean(np.power(10, spl_arraylike/10)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0e58cadf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start with the interval  1D\n"
     ]
    }
   ],
   "source": [
    "average_interval = \"1D\"\n",
    "print (\"Start with the interval \", average_interval)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3c6f0960",
   "metadata": {},
   "outputs": [],
   "source": [
    "station_names = [\n",
    "    \"AATA\",\"AATB\",\"APTY\",\"ARNM\",\"ATHN\",\"BKSN\",\n",
    "    \"CALM\",\"DJON\",\"DOMB\",\"DOMC\",\"DRBS\",\"FSMT\",\n",
    "    \"HRMS\",\"INVK\",\"IRK2\",\"IRK3\",\"IRKT\",\"JBGO\",\n",
    "    \"JUNG\",\"JUNG1\",\"KERG\",\"KIEL\",\"KIEL2\",\"LMKS\",\n",
    "    \"MCRL\",\"MGDN\",\"MOSC\",\"MRNY\",\"MXCO\",\"NAIN\",\n",
    "    \"NANM\",\"NEWK\",\"NRLK\",\"OULU\",\"PSNM\",\"PTFM\",\n",
    "    \"PWNK\",\"ROME\",\"SNAE\",\"SOPB\",\"SOPO\",\"TERA\",\n",
    "    \"THUL\",\"TSMB\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d2cf6c10",
   "metadata": {},
   "outputs": [],
   "source": [
    "start_date = [ \"1\", \"1\", \"1960\", \"0\", \"0\"]\n",
    "end_date = [\"31\", \"12\", \"2019\", \"23\", \"59\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fd0c14a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AATAfully completed\n",
      "AATBfully completed\n",
      "APTYfully completed\n",
      "ARNMfully completed\n",
      "ATHNfully completed\n",
      "BKSNfully completed\n",
      "CALMfully completed\n",
      "DJONfully completed\n",
      "DOMBfully completed\n",
      "DOMCfully completed\n",
      "DRBSfully completed\n",
      "FSMTfully completed\n",
      "HRMSfully completed\n",
      "INVKfully completed\n",
      "http://nest.nmdb.eu/draw_graph.php?wget=1&stations[]=IRK2&tabchoice=1h&dtype=corr_for_pressure&output=ascii&date_choice=bydate&last_label=days_label&tresolution=60&force=0&start_day=1&start_month=1&start_year=2015&start_hour=0&start_min=0&end_day=31&end_month=12&end_year=2015&end_hour=23&end_min=59&yunits=0\n",
      "An exception occurred with IRK22015.csv\n",
      "IRK2fully completed\n",
      "http://nest.nmdb.eu/draw_graph.php?wget=1&stations[]=IRK3&tabchoice=1h&dtype=corr_for_pressure&output=ascii&date_choice=bydate&last_label=days_label&tresolution=60&force=0&start_day=1&start_month=1&start_year=2015&start_hour=0&start_min=0&end_day=31&end_month=12&end_year=2015&end_hour=23&end_min=59&yunits=0\n",
      "An exception occurred with IRK32015.csv\n",
      "IRK3fully completed\n",
      "IRKTfully completed\n",
      "JBGOfully completed\n",
      "JUNGfully completed\n",
      "JUNG1fully completed\n",
      "KERGfully completed\n",
      "KIELfully completed\n",
      "KIEL2fully completed\n",
      "LMKSfully completed\n",
      "MCRLfully completed\n",
      "MGDNfully completed\n",
      "MOSCfully completed\n",
      "MRNYfully completed\n",
      "MXCOfully completed\n",
      "NAINfully completed\n",
      "NANMfully completed\n",
      "NEWKfully completed\n",
      "http://nest.nmdb.eu/draw_graph.php?wget=1&stations[]=NRLK&tabchoice=1h&dtype=corr_for_pressure&output=ascii&date_choice=bydate&last_label=days_label&tresolution=60&force=0&start_day=1&start_month=1&start_year=2015&start_hour=0&start_min=0&end_day=31&end_month=12&end_year=2015&end_hour=23&end_min=59&yunits=0\n",
      "An exception occurred with NRLK2015.csv\n",
      "NRLKfully completed\n",
      "OULUfully completed\n",
      "PSNMfully completed\n",
      "PTFMfully completed\n",
      "PWNKfully completed\n",
      "ROMEfully completed\n",
      "SNAEfully completed\n",
      "SOPBfully completed\n",
      "SOPOfully completed\n",
      "TERAfully completed\n",
      "THULfully completed\n",
      "TSMBfully completed\n",
      "finished\n"
     ]
    }
   ],
   "source": [
    "# load all the stations\n",
    "for current_station in station_names:\n",
    "    for i in range(int(start_date[2]), int(end_date[2])+1):\n",
    "        station_file = Path(current_station+str(i)+'.csv')\n",
    "        if not station_file.exists():\n",
    "            URL = \"http://nest.nmdb.eu/draw_graph.php?wget=1&stations[]=\"+current_station+\"&tabchoice=1h&dtype=corr_for_pressure&output=ascii&date_choice=bydate&last_label=days_label&tresolution=60&force=0&start_day=\"+start_date[0]+\"&start_month=\"+start_date[1]+\"&start_year=\"+str(i)+\"&start_hour=\"+start_date[3]+\"&start_min=\"+start_date[4]+\"&end_day=\"+end_date[0]+\"&end_month=\"+end_date[1]+\"&end_year=\"+str(i)+\"&end_hour=\"+end_date[3]+\"&end_min=\"+end_date[4]+\"&yunits=0\"\n",
    "            print(URL)\n",
    "            try:\n",
    "                req = urllib.request.Request(URL)\n",
    "            #req.add_header('User-Agent', 'urllib/0.1')\n",
    "                with urllib.request.urlopen(req) as f:\n",
    "                    content = f.readlines()\n",
    "                    fromhere = False\n",
    "                    content_p = \"\"\n",
    "                    for line in content:\n",
    "                        if \"start_date_time\" in line.decode(\"utf-8\", \"strict\"):\n",
    "                            fromhere = True\n",
    "                        if fromhere:\n",
    "                            content_p =  content_p+ line.decode()\n",
    "                with station_file.open('wt') as fh:\n",
    "                    fh.write(content_p)\n",
    "                    fh.close\n",
    "                time.sleep(1)\n",
    "                #print(current_station+'completed ' + str(i))\n",
    "            except:\n",
    "                print(\"An exception occurred with \"+str(station_file))\n",
    "            #time.sleep(6)\n",
    "    print(current_station+'fully completed')\n",
    "print(\"finished\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2db1f566",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An exception occurred, IRK22015.csv\n",
      "An exception occurred, IRK32015.csv\n",
      "An exception occurred, NRLK2015.csv\n",
      "finished\n"
     ]
    }
   ],
   "source": [
    "# Populate data structures\n",
    "start_date = [ \"1\", \"1\", \"1960\", \"0\", \"0\"]\n",
    "end_date = [\"31\", \"12\", \"2019\", \"23\", \"59\"]\n",
    "combineddata = []\n",
    "for current_station in station_names:\n",
    "    listofcosmics = []\n",
    "    for i in range(int(start_date[2]), int(end_date[2])+1):\n",
    "        station_file = Path(current_station+str(i)+'.csv')\n",
    "        try:\n",
    "            readframe = (pd.read_csv(station_file,sep=';',skiprows=1,names = ['start_date_time', current_station],parse_dates=['start_date_time'], index_col=['start_date_time']))\n",
    "            #print(\"Opened \"+str(station_file))\n",
    "        except:\n",
    "            print(\"An exception occurred, \"+str(station_file))\n",
    "        listofcosmics.append(readframe)\n",
    "    listofcosmics[0] = pd.merge(listofcosmics[0],listofcosmics[1],  how='outer', left_index=True, right_index=True)\n",
    "    listofcosmics[0] = listofcosmics[0].fillna(0)\n",
    "    listofcosmics[0][current_station] = listofcosmics[0][str(current_station+\"_x\")] + listofcosmics[0][str(current_station+\"_y\")]\n",
    "    listofcosmics[0] = listofcosmics[0].drop(columns=[str(current_station+'_x'), str(current_station+'_y')])\n",
    "    howmanyitems = len(listofcosmics)\n",
    "    for i in range(2, howmanyitems):\n",
    "        listofcosmics[0] = pd.merge(listofcosmics[0],listofcosmics[i],  how='outer', left_index=True, right_index=True)\n",
    "        listofcosmics[0] = listofcosmics[0].fillna(0)\n",
    "        listofcosmics[0][current_station] = listofcosmics[0][str(current_station+\"_x\")] + listofcosmics[0][str(current_station+\"_y\")]\n",
    "        listofcosmics[0] = listofcosmics[0].drop(columns=[str(current_station+\"_x\"), str(current_station+\"_y\")])\n",
    "    combineddata.append(listofcosmics[0])\n",
    "print(\"finished\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "dab6b38c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['AATA', 'AATB', 'APTY', 'ARNM', 'ATHN', 'BKSN', 'CALM', 'DJON', 'DOMB', 'DOMC', 'DRBS', 'FSMT', 'HRMS', 'INVK', 'IRK2', 'IRK3', 'IRKT', 'JBGO', 'JUNG', 'JUNG1', 'KERG', 'KIEL', 'KIEL2', 'LMKS', 'MCRL', 'MGDN', 'MOSC', 'MRNY', 'MXCO', 'NAIN', 'NANM', 'NEWK', 'NRLK', 'OULU', 'PSNM', 'PTFM', 'PWNK', 'ROME', 'SNAE', 'SOPB', 'SOPO', 'TERA', 'THUL', 'TSMB']\n",
      "                        AATA\n",
      "start_date_time             \n",
      "2019-01-04 06:59:00  3775.33\n",
      "2019-01-04 07:59:00  3781.68\n",
      "2019-01-04 08:59:00  3773.55\n",
      "2019-01-04 09:59:00  3831.31\n",
      "2019-01-04 10:59:00  3784.70\n",
      "...                      ...\n",
      "2019-12-31 19:59:00  3756.13\n",
      "2019-12-31 20:59:00  3774.63\n",
      "2019-12-31 21:59:00  3747.15\n",
      "2019-12-31 22:59:00  3781.48\n",
      "2019-12-31 23:59:00  3761.70\n",
      "\n",
      "[8105 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "#check that thte sequences match\n",
    "print(station_names)\n",
    "print(combineddata[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e0a38083",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        TSMB\n",
      "start_date_time             \n",
      "1976-12-04 00:00:00  330.556\n",
      "1976-12-04 01:00:00  330.972\n",
      "1976-12-04 02:00:00  330.694\n",
      "1976-12-04 03:00:00  331.139\n",
      "1976-12-04 04:00:00  330.500\n",
      "...                      ...\n",
      "2019-11-01 19:00:00  332.000\n",
      "2019-11-01 20:00:00  332.139\n",
      "2019-11-01 21:00:00  331.583\n",
      "2019-11-01 22:00:00  331.639\n",
      "2019-11-01 23:00:00  332.000\n",
      "\n",
      "[335841 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "# random sample test of one\n",
    "print(combineddata[43])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "746c7442",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 17300 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 18496 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 16188 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot several stations to check it makes sense\n",
    "for station in range (0, 3):\n",
    " \n",
    "    combineddata[station].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9e20af18",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                    latitude longitude  depth   mag\n",
      "time                                                               \n",
      "1960-01-02 12:21:58+00:00            -55.877     -1.89     15   6.3\n",
      "1960-01-02 22:51:46.080000+00:00  35.5563333  -121.351      6  4.04\n",
      "1960-01-03 11:24:05+00:00               43.7    84.542     15   5.7\n",
      "1960-01-04 06:16:35+00:00             11.374    42.609     15   6.1\n",
      "1960-01-04 12:52:00+00:00             45.069    26.829     40   5.4\n",
      "...                                      ...       ...    ...   ...\n",
      "2019-12-31 19:02:55.645000+00:00     44.0786  150.2151     10   4.8\n",
      "2019-12-31 19:10:10.306000+00:00      24.074  121.6908  12.21   4.6\n",
      "2019-12-31 23:07:42.730000+00:00     17.9226  -66.8408      5   4.2\n",
      "2019-12-31 23:09:44.616000+00:00      9.9422  -84.4593  59.75     4\n",
      "2019-12-31 23:56:59.933000+00:00    -20.5382  168.9571  21.46   4.3\n",
      "\n",
      "[427535 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "# seismic data read in\n",
    "sfs = []\n",
    "start_year = 1960\n",
    "end_year = 2019\n",
    "types_dict = {'time':float, 'latitude':\"string\", 'longitude':\"string\", 'depth':\"string\", 'mag':\"string\"}\n",
    "for i in range(start_year, end_year+1):\n",
    "    sfs.append(pd.read_csv('siesmic-'+str(i)+'.csv',sep=',',skiprows=1,names = ['time', 'latitude', 'longitude', 'depth', 'mag'],parse_dates=['time'],index_col=['time'],usecols = [i for i in range(5)],dtype=types_dict))\n",
    "sframe= pd.concat(sfs)\n",
    "print(sframe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5fead4ec",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['latitude', 'longitude', 'depth', 'mag'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# manipulation with seismic data\n",
    "cols = sframe.columns\n",
    "print(cols)\n",
    "#cols.remove('time')\n",
    "sframe['latitude'] = pd.to_numeric(sframe['latitude'], errors='coerce')\n",
    "sframe['longitude'] = pd.to_numeric(sframe['longitude'], errors='coerce')\n",
    "sframe['depth'] = pd.to_numeric(sframe['depth'], errors='coerce')\n",
    "sframe['mag'] = pd.to_numeric(sframe['mag'], errors='coerce')\n",
    "#print(sframe.dtypes)\n",
    "#sframe=pd.convert_objects(convert_numeric=True)\n",
    "#for col in cols:\n",
    "#    sframe[col] = sframe[col].astype(float)\n",
    "#sframe.index.tz_convert(None)\n",
    "#sframe.tz_localize(None)\n",
    "sframe_notz = sframe.tz_convert(None)\n",
    "daily_quakes = sframe_notz.resample(average_interval, origin='1960-01-01').apply(lambda spl: 10*np.log10(np.mean(np.power(10, spl/10))))\n",
    "daily_quakes = daily_quakes.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2bdfb7a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='time'>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 17732 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the quake data\n",
    "sframe['mag'].plot()\n",
    "daily_quakes['mag'].plot()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3cc505e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start Sunspots!\n"
     ]
    }
   ],
   "source": [
    "print (\"Start Sunspots!\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4a418246",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the sunspot data\n",
    "sun_file = Path('sunspots.csv')\n",
    "content_p = \"\"\n",
    "if not sun_file.exists():\n",
    "    URL = \"http://cesar.kso.ac.at/sunspot_numbers/daily_sn.csv\"\n",
    "    print(URL)\n",
    "    req = urllib.request.Request(URL)\n",
    "    req.add_header('User-Agent', 'urllib/0.1')\n",
    "    with urllib.request.urlopen(req) as f:\n",
    "        content = f.readlines()\n",
    "        for line in content:\n",
    "            content_p =  content_p+ line.decode()\n",
    "    with sun_file.open('wt') as fh:\n",
    "        fh.write(content_p)\n",
    "        fh.close"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "79bbe9b8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            tot    sn    ss   diff\n",
      "time                              \n",
      "1944-06-05   23    0   -23   -23  \n",
      "1944-06-13    0    0     0     0  \n",
      "1944-06-14    0    0     0     0  \n",
      "1944-06-16    0    0     0     0  \n",
      "1944-06-21    0    0     0     0  \n",
      "...         ...   ...   ...    ...\n",
      "2021-06-29   61   10   -51     -41\n",
      "2021-06-30   69   21   -48     -26\n",
      "2021-07-01   53   11   -42     -30\n",
      "2021-07-02   67   21   -46     -25\n",
      "2021-07-03   76   23   -53     -29\n",
      "\n",
      "[21022 rows x 4 columns]\n",
      "            tot    sn    ss   diff\n",
      "time                              \n",
      "1944-06-05   23    0   -23   -23  \n",
      "1944-06-13    0    0     0     0  \n",
      "1944-06-14    0    0     0     0  \n",
      "1944-06-16    0    0     0     0  \n",
      "1944-06-21    0    0     0     0  \n",
      "...         ...   ...   ...    ...\n",
      "2021-06-29   61   10   -51     -41\n",
      "2021-06-30   69   21   -48     -26\n",
      "2021-07-01   53   11   -42     -30\n",
      "2021-07-02   67   21   -46     -25\n",
      "2021-07-03   76   23   -53     -29\n",
      "\n",
      "[21022 rows x 4 columns]\n",
      "tot      int64\n",
      "sn      string\n",
      "ss      string\n",
      "diff    string\n",
      "dtype: object\n",
      "tot       int64\n",
      "sn      float64\n",
      "ss      float64\n",
      "diff    float64\n",
      "dtype: object\n",
      "             tot    sn    ss  diff\n",
      "time                              \n",
      "1944-06-05  23.0   0.0 -23.0 -23.0\n",
      "1944-06-06   NaN   NaN   NaN   NaN\n",
      "1944-06-07   NaN   NaN   NaN   NaN\n",
      "1944-06-08   NaN   NaN   NaN   NaN\n",
      "1944-06-09   NaN   NaN   NaN   NaN\n",
      "...          ...   ...   ...   ...\n",
      "2021-06-29  61.0  10.0 -51.0 -41.0\n",
      "2021-06-30  69.0  21.0 -48.0 -26.0\n",
      "2021-07-01  53.0  11.0 -42.0 -30.0\n",
      "2021-07-02  67.0  21.0 -46.0 -25.0\n",
      "2021-07-03  76.0  23.0 -53.0 -29.0\n",
      "\n",
      "[28153 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "# data manipulations with sunspots\n",
    "sunframe = []\n",
    "sun_types_dict = {'time':float, 'total':\"string\", 'sn':\"string\", 'ss':\"string\", 'diff':\"string\"}\n",
    "sunframe.append(pd.read_csv(sun_file,sep=';',skiprows=0,names = ['time', 'tot', 'sn', 'ss', 'diff'],parse_dates=['time'],index_col=['time'],dtype=sun_types_dict))\n",
    "sundf= (pd.read_csv(sun_file,sep=';',skiprows=0,names = ['time', 'tot', 'sn', 'ss', 'diff'],parse_dates=['time'],index_col=['time'],dtype=sun_types_dict))\n",
    "\n",
    "print(sunframe[0])\n",
    "print(sundf)\n",
    "print(sundf.dtypes)\n",
    "sundf['sn'] = pd.to_numeric(sundf['sn'], errors='coerce')\n",
    "sundf['ss'] = pd.to_numeric(sundf['ss'], errors='coerce')\n",
    "sundf['diff'] = pd.to_numeric(sundf['diff'], errors='coerce')\n",
    "print(sundf.dtypes)\n",
    "\n",
    "daily_sunspots = sundf.resample(average_interval, origin='1960-01-01').mean()\n",
    "\n",
    "print(daily_sunspots)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fd304ce1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='time'>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 37180 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the sunspot progress\n",
    "daily_sunspots['tot'].plot()\n",
    "daily_sunspots['sn'].plot()\n",
    "daily_sunspots['ss'].plot()\n",
    "daily_sunspots['diff'].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "15486960",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='time'>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 23076 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show sunspot cycle\n",
    "daily_sunspots['tot'].plot()\n",
    "daily_quakes['mag'].plot()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3ee8722",
   "metadata": {},
   "source": [
    "If we wanted to add geomagnetic data we could get it from here: \n",
    "https://www.swpc.noaa.gov/products/goes-magnetometer#:~:text=GOES%20magnetometer%20data%20have%20been,during%20geomagnetic%20storms%20and%20substorms.&text=The%20data%20have%20often%20been,decisions%20for%20research%20sounding%20rockets. \n",
    "The GOES sats have info from 1996 onwards, but I don't understand the meaning of it!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "757a9bc6",
   "metadata": {},
   "source": [
    "next stage:\n",
    "Build a variable average filter for each/all data sets\n",
    "average from 1 hour data to n hours\n",
    "correlation function (i.e. dataset a to dataset b)\n",
    "variable window + offset function (i.e. compare dataset a, offset 1 week, with dataset b, no offset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2393d3dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe2faa25eb0>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 37580 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#daily_sunspots[\"tot\"][\"1980-01-01\":\"2019-01-01\"].plot()\n",
    "#daily_quakes['mag'] = 50 * daily_quakes['mag']\n",
    "#daily_quakes['mag'][\"1980-01-01\":\"2019-01-01\"].plot()\n",
    "#print(sframe.corrwith(sundf))\n",
    "#print(sunframe[0].corrwith(sframe['mag']))\n",
    "\n",
    "ax = daily_quakes['mag'][\"1975-01-01\":\"2019-01-01\"].plot(title='Sunspots (number) vs Earthquakes (average daily magnitude)')\n",
    "#daily_raycount[\"1975\":\"2019\"].plot(ax=ax1)\n",
    "\n",
    "#ax.plot(daily_quakes['mag'][\"1975\":\"2019\"])\n",
    "ax1 = ax.twinx()\n",
    "ax1.plot(daily_sunspots[\"tot\"][\"1975-01-01\":\"2019-01-01\"], color='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3606fbb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.12402736865827761"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# do statistical correlation of sunspots and earthquakes (zero offset)\n",
    "daily_sunspots[\"tot\"][\"1975-01-01\":\"2018-01-01\"].corr(daily_quakes['mag'][\"1975-01-01\":\"2018-01-01\"], method='pearson')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "dc1a5805",
   "metadata": {},
   "outputs": [],
   "source": [
    "# offset hunter 1 year  - we look +/- 180 days either side of the 'day' to see how it changes correlation\n",
    "# e.g. if earthquakes happened more frequently 7 days after a sunspot peak, we should see it\n",
    "# and if sunspots happen more frequently 15 days after an earthquake, this would show up too\n",
    "correl_matrix=[]\n",
    "for var_offset in range (-180, 180):\n",
    "        offset_ds = daily_sunspots[\"tot\"][\"1975-01-01\":\"2018-01-01\"].shift(periods=var_offset)\n",
    "        #correl_matrix[monthoffset,dayoffset] = daily_quakes['mag'][\"1980-01-01\":\"2019-01-01\"].corr(offset_ds, method='pearson')\n",
    "        correl_matrix = correl_matrix + [(daily_quakes['mag'][\"1975-01-01\":\"2019-01-01\"].corr(offset_ds, method='pearson'))]\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d1ea606a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26700 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Here we plot the general correlation vs offset shift (0 is same day)\n",
    "#print(correl_matrix)\n",
    "#data = np.array([\n",
    "#correl_matrix\n",
    "#])\n",
    "#x, y = data.T\n",
    "offset_range = []\n",
    "offset_range.extend(range(-180,180))\n",
    "#print(offset_range)\n",
    "plot.title(\"Sunspot vs Earthquake Correlation, offset from -180 to +180 days, 1975-2019\")\n",
    "plot.xlabel(\"Offset (days)\")\n",
    "plot.ylabel(\"Correlation Coef\")\n",
    "plot.scatter(offset_range,correl_matrix[:])\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "86f484cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#offset hunter 1985-2010\n",
    "# looking for the same thing in a different part of the dataset\n",
    "correl_matrix_sm=[]\n",
    "for var_offset in range (-180, 180):\n",
    "        offset_ds = daily_sunspots[\"tot\"][\"1985-01-01\":\"2010-01-01\"].shift(periods=var_offset)\n",
    "        #correl_matrix[monthoffset,dayoffset] = daily_quakes['mag'][\"1980-01-01\":\"2019-01-01\"].corr(offset_ds, method='pearson')\n",
    "        correl_matrix_sm = correl_matrix_sm + [(daily_quakes['mag'][\"1985-01-01\":\"2010-01-01\"].corr(offset_ds, method='pearson'))]\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "90901f46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 27528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the different range\n",
    "offset_range = []\n",
    "offset_range.extend(range(-180,180))\n",
    "#print(offset_range)\n",
    "plot.title(\"Sunspot vs Earthquake Correlation, offset from -180 to +180 days, 1985-2010\")\n",
    "plot.xlabel(\"Offset (days)\")\n",
    "plot.ylabel(\"Correlation Coef\")\n",
    "plot.scatter(offset_range,correl_matrix_sm[:])\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b835434d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# offset hunter 10k days (takes a long time to run)\n",
    "# we also calculate p values, a large p = low statistical/confidence significance\n",
    "# a very low p = high confidence interval\n",
    "# (we want a high correlation and a low p for something 'useful')\n",
    "correl_matrix_smt = []\n",
    "corr_p = []\n",
    "for var_offset in range (-10000, 10000):\n",
    "    offset_ds = daily_sunspots[\"tot\"][\"1975\":\"2019\"].shift(periods=var_offset)\n",
    "    #correl_matrix[monthoffset,dayoffset] = daily_quakes['mag'][\"1980-01-01\":\"2019-01-01\"].corr(offset_ds, method='pearson')\n",
    "    correl_matrix_smt = correl_matrix_smt + [(daily_quakes['mag'][\"1975\":\"2019\"].corr(offset_ds, method='pearson'))]\n",
    "    corr_p = corr_p + [(daily_quakes['mag'][\"1975\":\"2019\"].corr(offset_ds, method=pearsonr_pval))]        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "78aecb32",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26528 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot large scale offsets (correlation and p values)\n",
    "offset_range = []\n",
    "offset_range.extend(range(-10000,10000))\n",
    "#print(offset_range)\n",
    "plot.title(\"Sunspot vs Earthquake Correlation, offset from -10000 to +10000 days, 1975-2019\")\n",
    "plot.xlabel(\"Offset (days)\")\n",
    "plot.ylabel(\"Correlation Coef/p value\")\n",
    "plot.scatter(offset_range,correl_matrix_smt[:])\n",
    "plot.scatter(offset_range,corr_p[:])\n",
    "\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "11548c3d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Maximum correlation coef = 0.3318872530602409\n",
      "Lowest P value =  1.9197353657904475e-235\n",
      "Offset for maximum correlation =  -5267\n",
      "Offset for lowest P value =  5267\n",
      "Offset in years = 14.420260095824778\n"
     ]
    }
   ],
   "source": [
    "# print a text summary of th emost significant values...\n",
    "print(\"Maximum correlation coef =\", max(correl_matrix_smt[:]))\n",
    "print(\"Lowest P value = \", min(corr_p[:]))\n",
    "print(\"Offset for maximum correlation = \", -correl_matrix_smt.index(max(correl_matrix_smt[:])))\n",
    "print(\"Offset for lowest P value = \", corr_p.index(min(corr_p[:])))\n",
    "print(\"Offset in years =\", (correl_matrix_smt.index(max(correl_matrix_smt[:]))/365.25))\n",
    "#years ago - max historical correlation\n",
    "#sunspots from 9 years ago = 35% correlation with high confidence earthquake intensity today..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ad5d54a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many stations in the dataset?\n",
    "#print(combineddata[21].columns.values)\n",
    "#columhead = combineddata[21].columns.values\n",
    "len(combineddata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e1447ee0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        AATA\n",
      "start_date_time             \n",
      "2019-01-04               NaN\n",
      "2019-01-05               NaN\n",
      "2019-01-06               NaN\n",
      "2019-01-07               NaN\n",
      "2019-01-08               NaN\n",
      "...                      ...\n",
      "2019-12-27       3767.550000\n",
      "2019-12-28       3763.804167\n",
      "2019-12-29       3764.147500\n",
      "2019-12-30       3751.696250\n",
      "2019-12-31       3769.609583\n",
      "\n",
      "[362 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "# megarayhunter - all stations; run the same thing across them to see if we see differences\n",
    "# doesn't work yet! - This is about where I'm at in development.\n",
    "correl_matrix_smt=[]\n",
    "corr_p =[]\n",
    "daily_raycount = []\n",
    "   \n",
    "for station in range(0,43):\n",
    "    combineddata[station] = combineddata[station].resample(average_interval, origin='1960-01-01').mean()\n",
    "    #print(combineddata[station])\n",
    "    daily_raycount.append(combineddata[station])\n",
    "print(daily_raycount[0][\"1975\":\"2019\"].shift(periods=10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "368ce9cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       BKSN\n",
      "start_date_time            \n",
      "2009-05-18              NaN\n",
      "2009-05-19              NaN\n",
      "2009-05-20              NaN\n",
      "2009-05-21              NaN\n",
      "2009-05-22              NaN\n",
      "...                     ...\n",
      "2019-12-27       123.888208\n",
      "2019-12-28       123.903042\n",
      "2019-12-29       123.562125\n",
      "2019-12-30       123.711542\n",
      "2019-12-31       123.954917\n",
      "\n",
      "[3880 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "print(daily_raycount[5][\"1975\":\"2019\"].shift(periods=10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "76526954",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AATA\n",
      "AATB\n",
      "APTY\n",
      "ARNM\n",
      "ATHN\n",
      "BKSN\n",
      "CALM\n",
      "DJON\n",
      "DOMB\n",
      "DOMC\n",
      "DRBS\n",
      "FSMT\n",
      "HRMS\n",
      "INVK\n",
      "IRK2\n",
      "IRK3\n",
      "IRKT\n",
      "JBGO\n",
      "JUNG\n",
      "JUNG1\n",
      "KERG\n",
      "KIEL\n",
      "KIEL2\n",
      "LMKS\n",
      "MCRL\n",
      "MGDN\n",
      "MOSC\n",
      "MRNY\n",
      "MXCO\n",
      "NAIN\n",
      "NANM\n",
      "NEWK\n",
      "NRLK\n",
      "OULU\n",
      "PSNM\n",
      "PTFM\n",
      "PWNK\n",
      "ROME\n",
      "SNAE\n",
      "SOPB\n",
      "SOPO\n",
      "TERA\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 70420 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "del offset_ds\n",
    "del correl_matrix_smf\n",
    "del corr_pf\n",
    "dayrange = 30\n",
    "day_set_range = []\n",
    "day_set_range.extend(range(-dayrange,dayrange))\n",
    "\n",
    "correl_matrix_smf = [[]]\n",
    "corr_pf = [[]]\n",
    "offset_ds = []\n",
    "cosmic_quake = [[0 for x in range (0,42)] for y in range (-dayrange, dayrange)]\n",
    "\n",
    "for station in range(0,42):   \n",
    "    print(station_names[station])\n",
    "    for var_offset in range (-dayrange, dayrange):\n",
    "        local_offset = []\n",
    "        local_offset = daily_raycount[station][\"1975\":\"2019\"].shift(periods=var_offset)\n",
    "        fs = pd.Series(local_offset[station_names[station]])\n",
    "        #correl_matrix_smf = correl_matrix_smf + \n",
    "        #print(var_offset, daily_quakes['mag'][\"1975\":\"2019\"].corr(fs, method='pearson'))\n",
    "        cosmic_quake[station][var_offset] = daily_quakes['mag'][\"1975\":\"2019\"].corr(fs, method='pearson')\n",
    "    #print(len(cosmic_quake))    \n",
    "    flagplot = pd.to_numeric(cosmic_quake[station][:], errors='coerce')\n",
    "    #flagplot.plot()\n",
    "    #print(len(flagplot[station]))\n",
    "    #plot.scatter(day_set_range,flagplot)\n",
    "    plot.plot(flagplot)\n",
    "    #plot.show()\n",
    "        #correl_matrix_smf[station] = correl_matrix_smf[station] + daily_quakes['mag'][\"1975\":\"2019\"].corr(fs, method='pearson')\n",
    "        #print(correl_matrix_smf) \n",
    "        #correl_matrix_smf[station] = correl_matrix_smf[station]+daily_quakes['mag'][\"1980\":\"2018\"].corr(fs, method='pearson')\n",
    "        #corr_pf[station] = corr_pf[station] + [daily_quakes['mag'][\"1975\":\"2019\"].corr(fs, method=pearsonr_pval)]    \n",
    "\n",
    "#print('offset', offset_ds[:])  \n",
    "#print('daily quakes', daily_quakes['mag'][:])\n",
    "#sporkquake = daily_quakes['mag'].to_frame()\n",
    "#print(sporkquake)\n",
    "#ts = pd.Series(sporkquake['mag'])\n",
    "#fs = pd.Series(offset_ds[station_names[1]])\n",
    "#print(type(sporkquake.to_series()))\n",
    "#print(type(offset_ds.to_series()))\n",
    "#print(sporkquake[\"1975\":\"2019\"])\n",
    "#print('offset again', offset_ds)\n",
    "#print('print ts',ts)\n",
    "#print('print fs', fs)\n",
    "#print(sporkquake.corr(offset_ds, method='pearson'))\n",
    "#correl_matrix_smf = ts[\"1975\":\"2019\"].corr(fs, method='pearson')\n",
    "#print(correl_matrix_smf)   \n",
    "\n",
    "#offset_range = []\n",
    "#offset_range.extend(range(-10000,10000))\n",
    "#print(offset_range)\n",
    "\n",
    "#plot.scatter(range (-30, 30),cosmic_quake[0][range (-30, 30)])\n",
    "#plot.scatter(offset_range,corr_p[:])\n",
    "\n",
    "#flagplot = pd.to_numeric(cosmic_quake[station], errors='coerce')\n",
    "#flagplot.plot()\n",
    "#plot.plot(flagplot)\n",
    "#plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "9ad53a96",
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "x and y must be the same size",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-146-09681caf96b8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moffset_range\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcorrel_matrix_smf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"Daily cosmics (THUL) correlated with average daily earthquake score (log-average/24h)\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscatter\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moffset_range\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcorr_pf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"p value for correlation, (pearson)\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlegend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Time shifted correlation between cosmic rays [THUL] and earthquakes'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/pyplot.py\u001b[0m in \u001b[0;36mscatter\u001b[0;34m(x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, plotnonfinite, data, **kwargs)\u001b[0m\n\u001b[1;32m   2893\u001b[0m         \u001b[0mverts\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverts\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0medgecolors\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0medgecolors\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2894\u001b[0m         \u001b[0mplotnonfinite\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mplotnonfinite\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2895\u001b[0;31m         **({\"data\": data} if data is not None else {}), **kwargs)\n\u001b[0m\u001b[1;32m   2896\u001b[0m     \u001b[0msci\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m__ret\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2897\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0m__ret\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/__init__.py\u001b[0m in \u001b[0;36minner\u001b[0;34m(ax, data, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1445\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1446\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mdata\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1447\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msanitize_sequence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1448\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1449\u001b[0m         \u001b[0mbound\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_sig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0max\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/cbook/deprecation.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*inner_args, **inner_kwargs)\u001b[0m\n\u001b[1;32m    409\u001b[0m                          \u001b[0;32melse\u001b[0m \u001b[0mdeprecation_addendum\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    410\u001b[0m                 **kwargs)\n\u001b[0;32m--> 411\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minner_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0minner_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    412\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    413\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/matplotlib/axes/_axes.py\u001b[0m in \u001b[0;36mscatter\u001b[0;34m(self, x, y, s, c, marker, cmap, norm, vmin, vmax, alpha, linewidths, verts, edgecolors, plotnonfinite, **kwargs)\u001b[0m\n\u001b[1;32m   4439\u001b[0m         \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mravel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4440\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4441\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"x and y must be the same size\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4442\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4443\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0ms\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: x and y must be the same size"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 4740 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot.scatter(offset_range,correl_matrix_smf[:], label=\"Daily cosmics (THUL) correlated with average daily earthquake score (log-average/24h)\")\n",
    "plot.scatter(offset_range,corr_pf[:], label = \"p value for correlation, (pearson)\")\n",
    "\n",
    "plot.legend()\n",
    "plot.title('Time shifted correlation between cosmic rays [THUL] and earthquakes')\n",
    "plot.xlabel(\"Timeshift (days)\")\n",
    "plot.ylabel(\"Correlation and p-value\")\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "552f2fc1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.356109919072931\n",
      "1.9058301817431175e-204\n",
      "3993\n",
      "3993\n",
      "0.21851890422627762\n",
      "-5209\n"
     ]
    }
   ],
   "source": [
    "print(min(correl_matrix_smf[:]))\n",
    "print(min(corr_pf[:]))\n",
    "print(10000-correl_matrix_smf.index(min(correl_matrix_smf[:])))\n",
    "print(10000-corr_pf.index(min(corr_pf[:])))\n",
    "print(max(correl_matrix_smf[:]))\n",
    "print(10000-correl_matrix_smf.index(max(correl_matrix_smf[:])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "23024181",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19480 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "correl_matrix_smf = []\n",
    "corr_pf = []\n",
    "for var_offset in range (-5000, 5000):\n",
    "    offset_ds = daily_raycount[\"1999\":\"2019\"].shift(periods=var_offset)\n",
    "    correl_matrix_smf = correl_matrix_smf + [daily_quakes['mag'][\"1999\":\"2019\"].corr(offset_ds['MXCO'], method='pearson')]\n",
    "    corr_pf = corr_pf + [daily_quakes['mag'][\"1999\":\"2019\"].corr(offset_ds['MXCO'], method=pearsonr_pval)]    \n",
    "#print(correl_matrix_smt)  \n",
    "\n",
    "offset_range = []\n",
    "offset_range.extend(range(-5000,5000))\n",
    "#print(offset_range)\n",
    "\n",
    "plot.scatter(offset_range,correl_matrix_smf[:])\n",
    "plot.scatter(offset_range,corr_pf[:])\n",
    "\n",
    "plot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "9fd372df",
   "metadata": {},
   "outputs": [],
   "source": [
    "correl_matrix_smft = []\n",
    "corr_pft = []\n",
    "for var_offset in range (-10000, 10000):\n",
    "    offset_ds = daily_raycount[\"1975\":\"2019\"].shift(periods=var_offset)\n",
    "    correl_matrix_smft = correl_matrix_smft + [sundf[\"tot\"][\"1975\":\"2019\"].corr(offset_ds['MXCO'], method='pearson')]\n",
    "    corr_pft = corr_pft + [sundf[\"tot\"][\"1975\":\"2019\"].corr(offset_ds['MXCO'], method=pearsonr_pval)]    \n",
    "#print(correl_matrix_smt)  \n",
    "\n",
    "offset_range = []\n",
    "offset_range.extend(range(-10000,10000))\n",
    "#print(offset_range)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "d247e560",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46352 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.7123943441942397\n",
      "8707\n"
     ]
    }
   ],
   "source": [
    "\n",
    "plot.scatter(offset_range,correl_matrix_smft[:],marker='.',  label=\"Daily cosmics (THUL) correlated with daily sunspot count\")\n",
    "plot.scatter(offset_range,corr_pft[:], marker='.', label = \"p value for correlation, (pearson)\")\n",
    "plot.legend()\n",
    "plot.title('Time shifted correlation between cosmic rays and sunspots')\n",
    "plot.xlabel(\"Timeshift (days)\")\n",
    "plot.ylabel(\"Correlation and p-value\")\n",
    "plot.show()\n",
    "\n",
    "print(min(correl_matrix_smft[:]))\n",
    "print(10000-correl_matrix_smft.index(min(correl_matrix_smft[:])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "a6913936",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:title={'center':'Daily earthquakes'}, xlabel='time'>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 21556 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "daily_quakes['mag'][\"1975\":\"2019\"].plot(title='Daily earthquakes')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "e20cd960",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='time'>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 11764 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "hour_quakes = sframe_notz.resample('1H', origin='1975-01-01').apply(lambda spl: 10*np.log10(np.mean(np.power(10, spl/10))))\n",
    "hour_quakes['mag'][\"1975\":\"2019\"].plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35181e89",
   "metadata": {},
   "outputs": [],
   "source": [
    "#morning to do\n",
    "#add labels to all plots\n",
    "#run for all cosmic ray stations\n",
    "#extend seismic data to all events (not just cut off at 4)\n",
    "#conduct dimensionless analysis (same thing with delta variable in time from frame-frame)\n",
    "#change the timebase for analysis - 1h, 3h, 6h, 12h, 24h, 5d, 7d, 1m\n",
    "#quality score for each cosmic ray station (period on, amount of data collected - as a rank)\n",
    "#random series test correlation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "924ba397",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x12742eaf0>]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 29312 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#daily_quakes['mag'] = 1/2 * daily_quakes['mag']\n",
    "ax = daily_quakes['mag'][\"1975\":\"2019\"].plot(title='Daily earthquakes')\n",
    "#daily_raycount[\"1975\":\"2019\"].plot(ax=ax1)\n",
    "\n",
    "#ax.plot(daily_quakes['mag'][\"1975\":\"2019\"])\n",
    "ax1 = ax.twinx()\n",
    "ax1.plot(daily_raycount[\"1975\":\"2019\"], color='red')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "e7013e5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x17740e0a0>"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 19980 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot at 0 intersection (day/day)\n",
    "plot.scatter(daily_quakes['mag'][\"1990\":\"2019\"],daily_raycount[\"1990\":\"2019\"], marker='.',s=0.1, label=\"Daily cosmics (THUL) vs Earthquake magnitude\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "db2de0da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x16a475cd0>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 17404 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot at -5000 days intersection\n",
    "offset_ds_plot = daily_raycount[\"1990\":\"2019\"].shift(periods=-4913)\n",
    "plot.scatter(daily_quakes['mag'][\"1990\":\"2019\"],offset_ds_plot, marker='.', s=0.1,label=\"Daily cosmics (THUL) vs Earthquake magnitude\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "1c56885f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       MXCO\n",
      "start_date_time            \n",
      "1990-01-01              NaN\n",
      "1990-01-02              NaN\n",
      "1990-01-03              NaN\n",
      "1990-01-04              NaN\n",
      "1990-01-05              NaN\n",
      "...                     ...\n",
      "2018-12-27       216.433909\n",
      "2018-12-28       216.217667\n",
      "2018-12-29       215.789875\n",
      "2018-12-30       215.544792\n",
      "2018-12-31       216.536957\n",
      "\n",
      "[10592 rows x 1 columns]\n",
      "time\n",
      "1990-01-01    4.993210\n",
      "1990-01-02    4.835273\n",
      "1990-01-03    4.509206\n",
      "1990-01-04    4.700446\n",
      "1990-01-05    4.823279\n",
      "                ...   \n",
      "2018-12-27    4.570452\n",
      "2018-12-28    4.559079\n",
      "2018-12-29    4.673080\n",
      "2018-12-30    4.510135\n",
      "2018-12-31    4.512682\n",
      "Freq: D, Name: mag, Length: 10592, dtype: float64\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 17720 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot at -5000 days intersection (positive correlation - max future)\n",
    "offset_ds_plot = daily_raycount[\"1990\":\"2018\"].shift(periods=8882)\n",
    "plot.scatter(daily_quakes['mag'][\"1990\":\"2018\"],offset_ds_plot, marker='.',s=0.1, label=\"Daily cosmics (THUL) vs Earthquake magnitude\")\n",
    "\n",
    "print(offset_ds_plot)\n",
    "print(daily_quakes['mag'][\"1990\":\"2018\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "bfbaf35b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       MXCO\n",
      "start_date_time            \n",
      "1992-01-01              NaN\n",
      "1992-01-02              NaN\n",
      "1992-01-03              NaN\n",
      "1992-01-04              NaN\n",
      "1992-01-05              NaN\n",
      "...                     ...\n",
      "2016-12-27       224.173625\n",
      "2016-12-28       225.840208\n",
      "2016-12-29       226.912375\n",
      "2016-12-30       225.899125\n",
      "2016-12-31       222.014833\n",
      "\n",
      "[9132 rows x 1 columns]\n",
      "              tot    sn     ss   diff\n",
      "time                                 \n",
      "1992-01-01  240.0   0.0 -240.0 -240.0\n",
      "1992-01-02  241.0  39.0 -203.0 -164.0\n",
      "1992-01-03  301.0  25.0 -277.0 -251.0\n",
      "1992-01-04  310.0  69.0 -241.0 -171.0\n",
      "1992-01-05  338.0  85.0 -253.0 -167.0\n",
      "...           ...   ...    ...    ...\n",
      "2016-12-27   27.0  27.0    0.0   27.0\n",
      "2016-12-28   11.0  11.0    0.0   11.0\n",
      "2016-12-29   11.0  11.0    0.0   11.0\n",
      "2016-12-30   11.0  11.0    0.0   11.0\n",
      "2016-12-31   41.0  21.0  -21.0    0.0\n",
      "\n",
      "[9132 rows x 4 columns]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1771fbd60>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 20180 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot at -5000 days intersection (positive correlation - max future)\n",
    "offset_ds_plotf = daily_raycount[\"1992\":\"2016\"].shift(periods=2000)\n",
    "print(offset_ds_plotf)\n",
    "print(daily_sunspots[\"1992\":\"2016\"])\n",
    "plot.scatter(daily_sunspots['tot'][\"1992\":\"2016\"],offset_ds_plotf, marker='.',s=0.1, label=\"Daily cosmics (THUL) vs Earthquake magnitude\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aceb4dc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
