Exploratory Data Analysis with the Binance API using Python and Pandas | MLTrader EP1

12.790 visualizações10 de jun. de 2

Transcrição what’s happening guys my.

    Name is.

    Nicholas renate and in this video we’re going to be taking a look at exploratory data analysis using the binance api so a lot of you have been asking in the comments below how can we start performing machine learning and deep learning using a real-time data pool well this is sort of the first step in that.

    Now this is going to be part of a larger series which i’m going to call the ml trader series and it’s really going to be driven by you so if there’s something that you want to see in terms of building up a machine learning trading bot or down the machine learning finance path i want to hear about it in the comments below so hit me up in the comments below and let me know what you’d like to see but without further ado let’s take a deeper look as to what we’ll be going through so in this video we’re going to cover four key things so first up what we’ll do is we’ll set up the binance api so you.

    Need to set up a binance account and grab an api key reasonably straightforward to do then what we’re going to do is we’re actually going to pull down the data from binance with a python library called python binance makes it super easy to grab this data down and then we’re going to transition to the data science phase so we’re going to bring that data and convert it into a pandas data frame so we’ll be able to traverse it work with it transform it as we like and then last but.

    Not least we’ll do a little bit of visualization using a matplotlib finance library so we’ll actually be able to visualize the candlesticks without historical data.

    Now let’s take a look as to how the store gonna work so again this is gonna be super interactive so what we’re gonna do is we’re first i’m gonna set up an api key on binance so again all you.

    Need to do is have a binance account and you should be able to grab an api key i’ll show you how to do that then what we’re going to do is we’re going to install a library inside of our jupyter.

    Notebook instance and again if you’re working inside of an ide so say pycharm or vs code again you can still do this you just.

    Need to install it at the command prompt.

    No biggie then what we’re going to do is we’re going to explore and visualize our data using a visualization library called mpl finance so this is a finance library built on top of matplotlib so it just makes it easier to work with finance data alrighty on that.

    Note ready to do it let’s get to it all righty guys let’s go on ahead and get this up and running so in order to go through our binance eda we’re gonna have a seven q steps that we’re going to be going through so first up what we’re going to.

    Need to do is set up let’s zoom in or make sure we’re zoomed in yep 125.

    cool so first up what we’re going to do is set up our binance account and specifically what we’re going to do there is grab our api keys then what we’re going to do is authenticate so this is really just grabbing those api keys and authenticating to the binance client then we’re going to grab some data so we’re going to get our tickers so you’re able to grab all the tickers from binance we’re going to grab our market depth for a specific ticker so this sort of shows you what the orders are in the pipeline then we can grab some historical data and this would specifically be the type of data that you use to train a bot so you use the historical data to be able to teach your machine learning agent as to when a stock is likely to go up or down and so on and so forth then what we’re going to do is pre-process that historical data so by default when we actually grab this out of the python client the formatting isn’t all that great in order to work with it from a data science perspective so we’ll do a little bit of pre-processing on that so specifically we’ll convert our date columns and convert our.

    Numeric columns and then last but.

    Not least what we’re then going to do is perform a little bit of visualization so we can actually go on ahead and take a look at our data so first things first what we.

    Need to do is go on ahead to binance and get an api key.

    Now i’ve already gone and created an account all i went and did is went and registered so again pretty straightforward so say for example i open up an incognito window if we go to binance all you.

    Need to do is just go on ahead and hit register fill out your details and then you’ll get a link sent to your email you just.

    Need to verify your account you should be good to go and then all you.

    Need to do is go on ahead and log in.

    Now once you log in it’ll look a little bit like this again i don’t really use binance for any trading or crypto stuff that i’m doing but it sort of looks a little bit like this when you actually log in.

    Now what we.

    Need to do is set up our api key so let’s go ahead and do that so to set up our api key you just.

    Need to go on ahead to this little button over here so let me zoom in so over here you can see that you’ve got this profile button so if you hover over a profile what you’re going to get down here is api management so api management is what’s going to allow you to create your api as you might have guessed so let’s go ahead select api management and you can see i’ve already got one api created.

    Now what we can do is create a.

    New api up here so all you.

    Need to do in order to do that is hit label the api key or label the api key and then hit create api so let’s create one so let’s call it um i don’t know finance eda and then hit create api and this is going to ask you to verify so i’m just going to quickly verify and once we’re verified you’ll get your api key details so from here you’ll get two key things you’ll get this api key which you’ll.

    Need and you’ll also get a secret key which you’ll also.

    Need.

    Now what i’m going to suggest in order to make this a little bit more secure is that you edit the restrictions so what i’ll get you to do is hit edit restrictions from up here so what we’re going to do is take off the trading permissions and we’re also going to make it so that only your computer from your ip address can access it so what we can do is hit edit restrictions up here and then we’re going to untick enable spot and margin trading and then we’re down here we’re going to select restrict access to trusted ips only so down here you can see this so.

    Normally it’s set to unrestricted we’re going to choose restrict access to trusted ips then what i want you to do is go on ahead and grab your ip address so this is pretty easy just go to google and go my ip address and you can see you’re going to get your ip address there just copy that.

    Now this may change so i was talking to someone previously about how this actually works so your ip address might change over time so you can request a static ip address from your internet service provider however it might change so whilst it might be this right.

    Now it might change a little bit later so you might.

    Need to update that restriction inside of binance but for.

    Now that’s how to grab your ip address so if we copy this and then paste this into this box down here and hit confirm that is our api key.

    Now set up and.

    Now a little bit more secure so then what we’re going to do is hit save and then enter our security verification again and that is our api key.

    Now set up so again reasonably straightforward so all you really.

    Need to do is go on ahead to the profile button hit api management create your api key and then it’s up to you whether or.

    Not you choose to change the restrictions but i find that it’s a little bit more secure particularly when you’re working with the training platform you want your stuff to be as secure as possible so what we’ll.

    Now do is we’re.

    Now going to copy our api key and under setup we’re just going to create a variable called api key and we’re going to set that equal to our api key and then we’re also going to set one up for our secret so if we copy this type in secret and then paste that in there so all we’re doing is we’re creating two string variables to hold these api details that we just went and pulled down so first up what i’ve written is api key equals and then this value over here is.

    No different to what you see up here and then i’ve written secret equals and then this value over here is.

    No different to what we’ve got over here so we’re just bringing those into our jupiter.

    Notebook then what we can do is run that cell and that is our api key and secret.

    Now done.

    Now the.

    Next thing that we.

    Need to do is actually go ahead and install a few dependencies so specifically what we’re going to be doing is installing the python binance library so if we type in python binance so this is the library that we’re actually going to be using to pull down our data so if you want to check out the documentation it’s all available here and as per usual all of the code that we write is going to be available in the description below via github and i’ll also include links to anything that i sort of mentioned in here so you can pick that up as well but again keep in mind this tutorial is going to be an interactive one so it’s going to be a bigger series so if you’ve got stuff that you want to see hit me up in the comments below guys i want to hear your thoughts i want to see where to take this ideally i’m thinking we’re probably going to end up building up a full hardcore machine learning trading bot but we’ll see where we go it’s up to you so what we are.

    Now going to do is install this so let’s go ahead and install our dependencies okay so i’ve gone and written one line of code there but we’re installing three dependencies so what i’ve written is exclamation mark pip install and then three libraries so first up it’s python binance so this is this library here and then what we’ve gone in installed is pandas so pandas is a really common data science library that helps us work with or well it’s really it’s a tabular data library but it just helps us work with data in a tabular format so again really really good for finance and particularly for financial analysis then the last library that we’ve gone and installed or we’re about to go and install is mpl finance so let’s take a look at that one mpl finance this is this over here so this just makes it easier to plot uh financial data so again you can do your candlesticks and you can do a whole bunch of different charts you can perform movie averages plot volume pretty cool so again it makes it super easy to actually plot this data so let’s go ahead and run this so let’s actually take a look at that full line of code so exclamation mark pip install python dash binance pandas and then mpl finance so let’s run that and that will install our dependencies cool so that is.

    Now done.

    Now the.

    Next thing that we.

    Need to do is actually import those dependencies so let’s import them okay those are our dependencies.

    Now imported so what i’ve gone and written there is from binance import client so this is going to be our client and then i’ve imported some additional dependencies to work with websockets later on but we’re.

    Not going to do that in this particular case so then the fur the.

    Next one is threaded websocket manager and again websocket is lowercase you would have seen a little error there it’s because i had the s as capitals should be lowercase and then the threaded depth cache manager so these are all our binance dependencies we’re mainly going to be working with client and then what i’ve gone and also written is import pandas pd so this kind of allows to work with pandas reasonably easily alrighty so that is our setup.

    Now done so again what we did there is we set up our api key and our secret we went and installed our dependencies and then we went and imported our dependencies so all well and good.

    Now the.

    Next thing that we.

    Need to do is just set up our client and actually authenticate to binance so let’s do it okay and it’s just a single line so again reasonably straightforward to get up and running with this so in order to authenticate we just.

    Need to write client also we’re going to create a.

    New client variable and then we’re going to set that equal to the client class which is what we imported from up here then to actually authenticate we just.

    Need to pass through two variables these are our api key and our secret so what we actually went and got from binance rps remember we got it from over here we created our two.

    New variables.

    Now in order to authenticate to the python client we just.

    Need to pass those to the client class so the full line is client equals client with an upper case and then pass through api key and secret and we are all well and good.

    Now what we can go and do is start grabbing some data so first up we’re going to grab some ticker data so we’ll just take a look at what’s available there.

    Now again this is going to be purely exploratory so we’re.

    Not going to do anything too hardcore this is really all about getting familiar with the binance api so let’s grab our tickers first up okay so those are our tickers.

    Now available so you can see here that by running this line of code we’re able to get tickers for virtually every single currency pair cryptocurrency pair that’s available inside of binance so pretty cool right like you can grab a bunch of stuff so if we wanted to grab i don’t know bitcoin and busd or what is that i don’t even know what that is so say for example we wanted to grab uh bitcoin and eth i believe that’s in there let’s search for it btc eth that’s one there should be another let’s go up oh it should be e t c b h e t h b t c there you go so again you can grab all of the different coins that you’ve got available in here.

    Now again i’m.

    Not a massive cryptocurrency expert but you can start to see that you’re able to get all of these tickers reasonably easily and so what you’re actually getting from here is both the symbol and the current price so if we actually take a look to grab a single one all we’d.

    Need to do is go into that so we can go tickers and then if we wanted to grab the first value so that’s the first ticker that we’re getting back so etc or eth btc.

    Now we can grab the second one just by accessing that second array and you can see that we’re grabbing the symbol and the price so if we wanted to grab the symbol you can just access it using indexing we can also grab the price as well there you go.

    Now you can see that this price is in a string so when we actually go and process this down here we’re actually going to perform a conversion to convert the string data that we’re getting back to a.

    Numeric value because as of right.

    Now you’re.

    Not going to be able to add strings together it’s.

    Not going to work as you want it to so you actually.

    Need to convert this to a.

    Numeric value let’s take a look at the line that we actually wrote to grab this data so we’ve written tickers we’re setting our variable equal to tickers and we’ve set that equal to client dot get all tickers so this get all tickers function allows you to go and grab all the ticker values.

    Now there are a whole heap of functions available inside of here so you can see that there is a bunch of stuff that you can actually do and if you type in get get is all really to do with getting different types of data from your account so you can get get your own trades get products get ticker get your trade fees get your withdrawal history so there’s a whole bunch of different types of data that’s actually available from the binance api uh.

    Now in this particular case we’re just grabbing our ticker data so we’ve written tickers equals client dot get all tickers and then we’ve got a parenthesis at the end because it is a function or method and then what we’ve gone and written below that is tickers to just go and render it out so if i just wrote this you wouldn’t actually see the result but we can also write a.

    New cell to show tickers cool right.

    Now that is done the.

    Next thing oh that’s btc usd there so that would be the usd currency conversion there pretty good to know again you’ve got a whole bunch of different types of currency pairs that are available in here.

    Now what we might want to do is actually put this inside of a pandas data frame so again if you’re familiar with data science or if you prefer working with pandas i personally think it just makes it a whole bunch easier to work with so let’s go ahead and do that okay that is our ticker data.

    Now in a data frame so again really really simple to actually put this together so all i’ve written is ticker underscore df and you could.

    Name this data frame really whatever you wanted to if you want it to be df or ticker df or whatever you want you can.

    Name that variable whatever you want so tica underscore df equals pd dot data frame data frame in caps or camel case and then to that we’re passing through our tickers data which we got from up here so if we.

    Now type in ticker df dot head you’ll actually be able to view the first five rows of data from your data frame and if we type in df.tail or tickerdf.tail we’ll be able to view the last five rows so again makes it way easier to actually process your data and work with it.

    Now we could even make our symbol an index and be able to search through it to make your life a whole bunch easier to actually work with this data so what we could do is we could type in ticker df dot set index and we could set that equal to symbol and type in in place equals true so then when we actually take a look at our ticker df data you can see that our symbol is.

    Now set as the index so if we wanted to grab a specific component just type in lock and then btc usd i think it was t there you go so that allows you to search through that data frame a whole lot easier so in this case to grab a specific currency pair we can actually just use the lock function to go and grab a specific location so you can see what i’ve written there is ticker underscore df dot set index and then basically what we’re saying is what column do we want to be our index or our main column and then we’ve gone and set that to our symbol column and then we’ve specified in place equals true so this means it’s going to apply it to the data frame in place then what we can actually do because that column is.

    Now set as the index we can actually start to search through it or identify a specific location a whole bunch easier so say for example we wanted to grab btc usd we can just type in ticker underscore df dot loc and then insider square brackets just pass through the currency pair that you want so in this case i’ve passed through btc usdt to grab the bitcoin usd conversion.

    Now we could type in what is it bt or eth btc to grab a different price again you can go and grab at different values.

    Now if you wanted the price all you can do is just type in price and this will give you the specific price.

    Now you can see here that this is a string so if you wanted to convert this you can just wrap it in a float and that will give you a.

    Numeric.

    Number so again a whole bunch of stuff that you can actually do with this data and reasonably quick to get up and running.

    Now that is our ticker data done so that’s step three done shows you how to grab some ticker data the.

    Next thing that we want to do is get some depth so say for example we wanted to grab marker depth for uh if you’re in bitcoin or the ethereum bitcoin pair let’s go ahead and do that okay that is our depth data.

    Now retrieved so what i’ve gone and written is depth equals client dot get underscore order underscore book and then we’ve passed through the symbol that we want so symbol equals e t h b t c.

    Now if we take a look at our depth data you can see that what we’re getting out of this is the last updated time and we’re also getting the different bids at the different price points so you can see here that at this particular bid this is the request this particular bid this is the particular request so again you can actually start searching through what’s actually happening here so again this allows you to get market depth pretty cool right.

    Now again you could change this if you wanted so say for example you wanted a bitcoin usd you could term btc usdt and again you’re going to be able to get your market depth and all the different prices as well as the different spread so again you can get the bid and ask you can see ask is down here cool right.

    Now what we can do is say for example we wanted to plot out our bids well what we can do is again put this inside of a data frame so let’s go ahead and do it okay there you go so what we’ve gone and done there is we’ve visualized just the bids.

    Not the asks in this particular case so what you can actually see here is that we’ve got the volume that we’re actually displaying the volume that we’re actually displaying here is the volume for the bids.

    Now we could change this to asks if we wanted to and that’s going to switch it up so again all you.

    Need to do is change that so let’s take a look at the code that we wrote so what we’ve got written is depth underscore df equals pd dot data frame data framing camel case and then to that we’ve passed through the data set that we want to visualize so in this case we’re visualizing the asks so depth and then inside of square brackets we’re passing through asks and then what we’re doing is we’re resetting our columns so if you don’t set the columns then you’re.

    Not actually going to get the column.

    Names displayed so if we actually drop this line you’d say that you’d have your data frame and column 1 would be.

    Name 0 column two would be.

    Named one again.

    No.

    Nothing crazy there and then what we’re going in doing is we’re visualizing the first five rows of data so again this is standard python functionality there so again first line is going and creating the data frame the second line is setting the column.

    Names and then third line is visualizing the actual data within that.

    Now again you could change this from asks to bids reasonably easily you just.

    Need to change the index that you’re passing through to this little section over here so that’ll change it to bids this is ask so again you can visualize this reasonably easily so this sort of shows you what’s possible with the binance api that being said what i really want to hear from you is where you’d like to see this taken so again are we building a full-blown machine learning trading bot are we looking at risk management are we looking at placing automated trades performing reinforcement learning do we want to visualize some of this stuff inside of a web app again this is all stuff that we could potentially do so that is step four.

    Now done so we’ve gone and grabbed our market depth.

    Now the.

    Next thing that we want to do is actually go ahead and grab some historical data.

    Now key thing to know is that when you’re working with these data sets it’s fine to take a look at them like this but the data isn’t actually in an appropriate format if you’d want to perform calculations on them i believe these two current data sets are currently saved a string so if i type in depth underscore df.d types you can see that right.

    Now they’re both reading as objects so price is an object volume is an object that means that this value and this value both of them are actually strings at the moment so in order to actually visualize and perform calculations you’d want to go and perform some data transformation that’s exactly what we’re going to do down here so in step five and six we’re going to grab our historical data for a particular currency pair we’ll then go and pre-process that historical data and then visualize it down here so let’s go on ahead and do this so we’re going to grab some historical data okay that is our historical data.

    Now retrieved so if we actually take a look at this historical variable we’ll actually be able to see what we’ve got historical so you can see we’ve got a whole bunch of.

    Numbers here.

    Now the unfortunate thing about this is that it doesn’t actually tell you what each one of these values are but i’ve actually gone and grabbed a bit of a dictionary that sort of shows that so these are what each one of these values represents so let me paste this into here so the first value is the open time and this is stored in epochs in seconds the.

    Next three values are your open high low close so open high low and close then you’ve got your volume value close time quote asset.

    Number.

    Number of trades take a buy base asset volume take a by quote asset volume.

    Now again i’m.

    Not too familiar with these so if you are a finance wizard let me know in the comments below what these are actually used for and then we’ve got a column which is just ignored.

    Now the line that we wrote to actually go and grab this is actually quite involved so what we wrote is historical equals client dot get underscore historical underscore clients and then to that we’ve passed through three different parameters so we’ve passed through the symbol that we want to grab so eth btc then we’ve passed through the interval that we want to grab our data for so in this case i’ve grabbed it at a daily interval and then i’ve also grabbed the date from that i want to grab it from.

    Now i’ve just passed through a really really early date i’m pretty sure eth wasn’t actually around back then it might have been who knows again let me know in the comments below when it was actually listed so i’ve got one january 2011.

   .

    Now let’s actually take a look at this function so here so client dot get historical clients so what you can see is that there are a whole bunch of different variables that you can pass to this so we can pass through the symbol we can pass through the type we can pass through the interval sorry the type is actually string so we can grab the symbol the interval the start date the end date the limit so how much data we want to grab and then we can also grab the spot or futures value so i believe by default what are we grabbing probably spot to be honest and then it looks like we’re going to grab back our open high low close and volume value so again a whole bunch of stuff that you can do with this if you wanted to change the interval you can do that as well so if i type in dot interval what is it klein so you’ve got a whole bunch of different intervals that you can pull through so say for example you wanted 15-minute data we can change that and grab our 15-minute data as well so again there’s a whole bunch of ways that you can actually manipulate this to get different types of data i’m guessing this is.

    Now going to take a while to grab back so hold on let’s wait for that to run all right i’m getting bored waiting for this let’s actually grab some let’s actually grab less data so rather than grabbing 15 minutes from the first of january 2011 let’s actually grab it from i don’t know yesterday so what would that be let’s grab it from a couple of days ago so 8th of june 2021 should be a little faster hopefully there you go so again grabbing a whole heap of data is obviously going to take a lot more time so i’ve passed through 15 minutes and tried to grab it back from the initial listing date gonna take it to under time to actually go back and grab that data.

    Now in this case all i did there was rather than going all the way back i’ve just changed it till a couple of days ago so the 8th of june 2021.

   .

    Now again if you wanted to grab all of that data back from a huge time period ago you could definitely do that so say for example we’re trying to build a bot that works a little bit better you might actually want to grab really really far out historical data in this case just to sort of show you what’s possible i’ve just cut down the date.

    Now again let’s go ahead and change this back to daily so if we type in klein and let’s grab the daily value so you can see that we’ve actually got client interval one day one hour one minute so on and so forth so you’ve got a whole bunch of different combos that you can pass through to here.

    Now i’m just gonna set the date way back just to make sure we grab everything okay that’s looking good so that’s our daily historical data back from first of january 2011.

   .

    Now i believe it’s limited to 500 values but you can start to see what we’ve got.

    Now what we want to do is as per usual get this data into a data frame but rather than just sort of leaving it at this stage we’re going to take this a bit further this time so let’s go on ahead and do that okay so again pretty straightforward to get our data into a data frame so i’ve just written hist underscore df equals pd dot data frame again in camel case and then we’ve passed through these data sets over here this data set so remember our historical data frame had all of these values.

    Now it had one two three four five six seven eight.

    Nine ten eleven twelve different values which correspond to this and again i’m gonna make this available inside of the jupyter.

    Notebook on the github description so again one two three four five six seven eight 9 10 11 12.

    so again 12 values 12 values these are the mappings for each one of those values so if we put this data into our data frame ideally we should get 12 different values so let’s go ahead and take a look at that so if we type in his df dot head you can see we’ve got all of our 12 values so starting at 0 we go all the way out to value 11.

   .

    Now what we can actually do is change those columns remember we changed our column.

    Names over here to price and volume for our depth data what we can do is a similar thing but this time we’ll do it for our historical data so let’s go ahead and do that so we can set our columns just by accessing the column attribute from our data frame so here’s to underscore df dot columns and this is going to allow us to reset our column.

    Names so let’s go on ahead and reset it so we’re just going to set our column.

    Names equal to these values over here so the open time open high low close volume so on and so forth so i’m just going to write this out and then we’ll take a look okay those are our column.

    Names.

    Now set.

    Now if we go and take a look at our data frame again you can see that we’ve.

    Now gone and set our column.

    Names.

    Now you might choose to make these couple a little bit smaller but i’ve gone and set them as they are so it’s all good so what we’ve gone and written is hist underscore df dot columns and then we’ve actually gone and set those equal to the different column.

    Names that we had from up here so i’ve just gone and passed through some examples so open time open high low close volume close time quote asset volume.

    Number of trades trader base what is it trader trader buy base volume traded by quote volume and then i’ll pass through the ignore column and then in order to visualize it we’ve just written hist underscore df.head so that gives us this thing over here.

    Now again you can view the bottom values so tail and that allows you to view the last five rows oh so it looks like we’ve pulled through more than 500 rows of data so this is all the way up to 1427 over here so you can see that which will mean we’ve actually got 1428 rows so if we actually take a look at his df dot shape you can see we’ve got 1 428 rows of data and we’ve got 12 columns so those 12 columns map to these the 1428 days of data means we’ve got 1 428 days of data so the rows map to dates okay.

    Now what was the.

    Next thing that we’re going to do oh yeah so right.

    Now the data types inside of this data frame aren’t all well and good so if we actually type in his df dot d types you can see that half of our columns or more than half of our columns are actually stored as objects so open high low close volume close time trader base volume or try to buy a base volume trade a by quote volume and ignore these are set as objects.

    Now object is just a fancy word for a string so right.

    Now they’re sort of strings which sucks if you’re trying to do financial analysis because you want these to be a.

    Numeric value so what we’re going to do is we’re going to pre-process this and actually fix this stuff up so again data pre-processing and data understanding is a core part of the data science life cycle and right.

    Now our data ain’t in the right format so we’ve got to go in here to fix this up so we’re going to convert our open time and our close time down here to date time types and we’re going to convert our open high low close volume and our quote asset volume.

    Number of trades is fine trader base try to buy base volume and trader buy quote volume we’re going to convert that or convert those to floats for.

    Now so let’s go on ahead and do this so first up let’s do our date so we’re going to convert open time and close time to date time values oh i should also know that we’re.

    Now into step six so we’ve done so that’s step five sort of done we’ve gone and grabbed our historical data we’re.

    Now up to step six where we’re going to pre-press it let’s actually add a couple of additional rows so we can see that so let’s go and do this okay that is our data or those are our date columns.

    Now updated so if we take in a look at our d types you should see that done so you can see that our open time and our close time are.

    Now date time 64 value so this means that they’re actually stored as dates so if we actually take a look at it by typing in dot head you can see that we’ve.

    Now got our different values.

    Now actually represented as a date time so you can see that we’ve got 14 15 what is that the 14th of july 2017 15th of july 2017.

    so on and so forth so it looks like our what is it eth btc only goes back to 2017 so that’s probably around about the date that it was listed.

    Now we can view the last five rows of data by typing in tail so again this goes back to yesterday so the 10th of june 2021 we can see our close time there as well cool so that is the first part of our data pre-processing done.

    Now you can see that all of our.

    Numeric values are still sort of string so we just got to go ahead and update those.

    Now so let’s do that so first up what we’re going to do is we’re going to create a list to represent all of the values that we want to pre-process so let’s do this first and then we’ll actually go and apply the transformation okay so those are our columns.

    Now stored in a list so what i’ve just gone and written there is.

    Numeric underscore columns equals and then i’ve passed through all of the object columns that are in our historical data frame as an object so it’s those ones that we actually want to go in ahead and convert them to.

    Numeric values so specifically i’ve gone and passed through open high low close volume so open high low close volume so these what is that five over there and then we’ve also got quote asset volume traded by base volume traded by quote volume so this value this value and this value so again i’ve skipped the.

    Number of trades because that’s already stored as an integer which you can see there so that’s all well and good we don’t.

    Need to change that one what we.

    Now.

    Need to do is actually go ahead and apply that transformation to convert it to a.

    Numeric value.

    Now so let’s do it so a cool thing to call that is if i wanted to grab all of those columns i can actually grab my historical data frame so here’s to underscore df pass through these.

    Numeric values and that’s going to filter the columns that have actually got available.

    Now by doing this what we’re actually going to do is we’re going to filter those columns and just apply this transformation on those set of columns okay that is our transformation.

    Now done so what i’ve gone and written is hist underscore df and then inside of square brackets.

    Numeric underscore columns and we’ve set that equal to again the same set of columns so again what we’re doing is we’re applying the transformation in place so here’s to underscore df and then again in square brackets.

    Numeric columns which are over these and then what we’ve gone and done is use the apply function or the apply method to go on ahead and convert these to.

    Numeric so to that we’ve passed through pd dot two.

    Numeric and then we specified access equals one to perform it on the actual columns so the full line is here’s to underscore df square brackets.

    Numeric columns equals hist underscore df square brackets.

    Numeric columns dot apply and then to that with pass through pd dot 2 underscore.

    Numeric and then set the access parameter equal to one.

    Now if we go and take a look at our data our data should actually be or these values should actually be.

    Numeric so again they’re still floats they’re still integers where they.

    Need to be but if we go and take a look at our d types they’ve all.

    Now been converted so you can see open high low close and volume these are.

    Now floats and our quote asset volume is a float our trader by base volume is float and our traded by quote volume is also.

    Now a float so that is our transformation.

    Now done.

    Now if we wanted to go and perform some statistical analysis we could actually grab our historical data frame dot describe and this will actually give us a whole bunch of statistical analysis so again you can grab the count the mean the standard deviation the minimum all your different quartiles and the max.

    Now again keep in mind this is a huge volume of data so you’re going to grab a whole or you’re going to be seeing a whole bunch of variants we can also type in i believe his.summary.

    Nope it’s.

    Not summer it’s info and this will give us a whole bunch of additional data on our data frame and whether or.

    Not we’re missing any values so again we can grab all the different data types that are actually stored in there the amount of memory that’s being used as well as all of our different columns and those types so again df.describe gives you all of your summary statistics so you’re able to grab the count mean standard deviation the minimum all your different quartiles and the max.

    Now keep in mind that this is only going to perform these summary statistics on your data that is.

    Numeric so it’s going to skip over your open time your close time and the ignore.

    Now if we say include equals object this will actually go and change the data type and actually go and perform those summary statistics on your object type columns but again this is.

    Not actually so we can also do it on our date time so let’s grab that so again.

    Now we’re going and grabbing our different data types on our columns so again you can do a whole bunch of analysis on this so if we go back to our summary statistics so again you can do a whole bunch of analysis on this you can play around with that analyze your data but again i want to hear where you want to take this so if there’s something specific that you’d like to see hit me up but we’re.

    Not done yet we’re going to go on ahead and visualize our data so the last thing that we’re going to do is under step 7 we’re going to go on ahead and visualize our different data sets.

    Now this is where we’re going to go on ahead and use that mpl finance library that i was telling you about so again we’re going to go on ahead and visualize our data to ideally produce something that looks a little bit like this so let’s go on ahead and do it so first up what we’re going to do is import the mpl finance libraries let’s do that okay so that is the mpl finance library.

    Now imported so what i’ve written there is import mpl finance as mpf and then what we can go ahead and do is actually visualize so let’s do this last thing okay that is our visualization.

    Now done so again i’ve kept it pretty simple in this particular case but we’ll extend this out in a second so what i’ve written is mpf dot plot and then to that we’ve passed through our data frame with an index set so specifically i’ve passed through his to underscore df dot set underscore index and then to that we’ve passed through the close time so this effectively means that we’re grabbing our data frame so let’s actually take look.set underscore index pass through close time so we’re doing that which basically means that we’re going to be sorting our data by the close time and then what we can actually do is take a look at the last hundred rows of data so tail 100 so by limiting the amount of data that we actually pass through to this particular visualization you’re actually able to see the candlesticks if we take off this tail value what you’ll see is that it’s really really small so again you might want to filter through how much data you actually visualize again if we and you can see that you’re actually getting this warning so warning you are plotting so much data that it may.

    Not be possible to see the details candles ohls c bars etc so if we actually limit the amount of data that we pass through it’s a little bit easier to see so you can start to see that you’re actually able to see the candles then but it doesn’t stop here so we can actually extend this out further so we can change the formatting so let’s do that so simply by passing through the type of chart that we want so in this case i’ve set it to candles there’s a bunch of different types and setting the style to charles you can see that we’re already changing the look and feel so we’re.

    Now getting a candlestick graph and we’ve gone and set some styling we can also go and choose to bring in some volume data so let’s do that and so that’s our volume.

    Now applied so in order to apply a volume all i’ve done is specified volume equals true so this allows you to visualize your volume in the same plot as well.

    Now we can also apply a title and we can also apply some moving averages so let’s do that and there you go so what we’ve then gone and done is specified a title parameter so title equals ethbtc last 100 days again you can change this to whatever you want so say for example we did last 120 days so we can change the tail value here to pull in different amounts of data then change the title as well and so you can see we’re.

    Now bringing in 120 days of data and we’ve changed the title up here and then we’ve gone and also applied some moving averages so you can see that we’ve got our moving average lines there as well.

    Now we can add in an additional moving average say we wanted a 30 day moving average all we.

    Need to do is pass through 30 to this value so actually to specify moving average i’ve just typed in mav equals and then inside of a tuple i’ve passed through the values that i want so this is going to be a 10 day moving average a 20 20-day moving average or if it wasn’t a daily data set it’d be 20 period and then i’ve gone and passed you a 30-day moving average as well so if we go and visualize that so you can.

    Now see that we’ve got a few different types of data so this will be 10 this will be 20 this will be 30.

    so this allows you to visualize your data from binance reasonably easily i really like this package so there’s a bunch of stuff that you can do with this this only sort of scratches the surface as to what you can actually pass through but that in a.

    Nutshell sort of shows you what’s actually possible with the binance api so that’s a little bit of exploratory data analysis for you.

    Now we’ve covered a bunch of stuff in this tutorial so what we’ve gone through it’s gone through setup how to get your api keys how to authenticate how to get ticker data how to get marker depth data how to get historical data as well as how to change your interval set and then last but.

    Not least i’ve shown how to actually go and pre-process your historical data to make it relevant for you and then we’ve gone and visualized using mpl finance which gives you a pretty chart like this and on that.

    Note that about wraps it up thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get.

    Notified of when i release future videos and again this is going to be a continuing series so if you’ve got feedback or if you want to see a particular type of thing or a particular type of implementation do hit me up in the comments below i really really want your feedback on this