Tag: Quotedata

Last time I left off at ditching Tradingview as a quote source. These things happen, and I learned quite a lot along the way, so I’m not troubled by it. Turning locally, as in my own hard drive, I began sifting through the data that gets archived by my chart provider.

Fortunately they aren’t super-protective of their quotes, so the files were essentially text data with some custom file extensions on the end. I began Python-ing a way to get these things filtered and saved in the proper folder for my Teardown mod.

The data is quite orderly:

Date, Time, Open, High, Low, Last, Volume, # of Trades, OHLC Avg, HLC Avg, HL Avg, Bid Volume, Ask Volume
2022/4/4, 13:22:45.0, 4561.50, 4561.50, 4561.50, 4561.50, 3, 3, 4561.50, 4561.50, 4561.50, 0, 3 

After a bit of filtering/detection, I had it doing things like this:

Last line in file: ESM22-CME-BarData.txt is:
2022/4/4, 20:32:40.0, 4571.25, 4571.25, 4571.25, 4571.25, 3, 3, 4571.25, 4571.25, 4571.25, 3, 0

Symbol: ES Last price: 4571.25 Volume: 3
File size in bytes: 12587 

(I’m counting filesize so I can wipe it when it hits a certain size, to keep search times down for some functions.)

One main task I had to figure out was filtering based on futures symbol. Most symbols follow the convention of:

 <symbol><monthcode><YY>

As with anything, there are a few exceptions where the year/month is flipped, but in aggregate my initial function seemed to do the trick.

I realized later on that this first pass wasn’t good enough – it wasn’t behaving EXACTLY as I needed, so it made sense to go through and refactor it into a function (or series of them), that worked how I wanted. Sometimes being “cute” and trying to do too many things at once creates technical debt that bites you in the ass when you least expect it.

After some pseudo-code restructuring to actually plan the flow of what I was trying to do, I came up with this result:

I set up a test where I threw it every symbol I had in my local directory to make sure it could handle things:

So far, so good! It was nice being able to chuck anything at it, even the dollar-sign prefix ones, and have it handle things in a sane way.

I now had to plan out the pseudo-code for making the main polling loop work. I’m not a Python expert by any means, so I just worked from some examples and adapted it to my purpose. After thinking a bit, I managed to get this flow:

This happens to me sometimes, I’m humming along and Python-ing my way to a given goal, and something gets thrown in my way that I didn’t quite expect.

Debugging isn’t glamorous, there aren’t any hacker-esque (in the hollywood sense) terminals with interesting things being scrolled/displayed/animated on them. Its just my boring Notepad++ open to a file and the shell where I’m running Python commands.

At one point, things were silently failing.

It could be my fault – I’m not an expert, as I said – so I could be structuring things too deeply in terms of logic IF .. THEN or just calling a bunch of functions like a fool. The result was I had to place some Debug statements like — print(“Did I even get to this point?”) in the code to figure out WHERE it was dying.

Not fun, in the least.

At this point, I’m nearly at the end of the Python part of implementing this mod — the rest of the steps are to get the voxel model encoded and saved to the proper directory, then in Teardown have it get read/decoded by a lua script.

So very close…. more to come.

There I was, puttering right along and getting some quote data when the unthinkable happened — Tradingview got wise to my quote-scraping ways, and the historical data I successfully was getting turned into error messages from the server. Dang.

Oh, cruel fate.

I had to regroup and try to salvage something. I had a bunch of regular expressions for filtering Tradingview’s price data, and I didn’t want them to go to waste! After a bit of thinking, I decided to do it the old fashioned way — scrape the Tradingview site directly.

This would require learning yet-another-skill, using a module called “Selenium” for Python. This clever library allows you to dive into page source code for stuff, or open up web pages and even simulate user “clicks” and data entry – for say, logging in.

I got to work, and soon I had something going:

browser.get(url)
browser.implicitly_wait(3)
## Have to use CSS selector when class names have spaces - replace with '.'
login_button_class = "tv-header__user-menu-button"
user_menu_class = "item-4TFSfyGO"
sign_in_email_class = "tv-signin-dialog__toggle-email"
user_name = "username"
browser.find_element(by=By.CLASS_NAME, value=login_button_class).click() # Click on user login
time.sleep(1)
browser.find_element(by=By.CLASS_NAME, value=user_menu_class).click() # Click on dropdown for email
time.sleep(1)
browser.find_element(by=By.CLASS_NAME, value=sign_in_email_class).click() # Click on email for User/Pass dialog
time.sleep(1)
input_username = browser.find_element(by=By.NAME, value=user_name).click() # Click on email for User/Pass dialog
pyautogui.typewrite(user) # works
pyautogui.press("tab") # Tab to next field
pyautogui.typewrite(password)
pyautogui.press("enter")
time.sleep(1)

There’s more setup involved prior to these actions, but I wanted to show the core of what I was doing. Clicking buttons, signing in, all of that. There was a way to cache the result — so I didn’t have to log in every single time, but for some reason that code didn’t work for me. I plowed ahead, undaunted.

A few bazillion log-ins later, and a bunch of “did you just log in from a new device” emails, I was at the page where you could add symbols to a watchlist, and it would helpfully display them on the right hand side of the page. I was set! (Or so I thought.)

No my friends, no such luck. Turns out the Tradingview chaps are quite resourceful. Let me explain.

In the “old days” you could look at a website’s source, it would have its data embedded in the page like “lastprice=46624.50”, which was trivial to scrape.

Well, websites are now reactive and do all kinds of things, which means what I was searching for was deep in the source. And I mean DEEP. Take a look at this relative path here:

/html/body/div[1]/div/div[1]/div[1]/div[1]/div[1]/div[2]/div/div[2]/div/div/div[2]/div/div[4]/div/div/span[1]/span

And that is just for ONE quote, mind you. (20-plus levels deep!)

Even if you got down there, Tradingview made sure to make it as hard as possible. How? Well, if you weren’t paying much attention, you’d pull up your watchlist and it would have some symbols with prices, like this:

DXY 98.62 BTCUSD 46639.25

So just dive down into the source and get it, right? Well, its more complicated than that. They don’t just display the prices in one go — oh no — some evil genius over there decided on any up/down tick to color a RANDOM portion of the quote green or red.

Which means a simple quote of:

46105.20

Turns into:

<span class="inner-ghhqKDrt">4610<span class="minus-ghhqKDrt">5.2</span></span>

So what, right?

It turns out that its monumentally harder to scrape a quote when the style of that quote changes on a whim. So, part of it is white, some of it is red/green at any given point. By splitting the quote apart in a random way, it turns out regular expressions that you’d use to grab it only get a fraction of the “normal” part:

4610 -- instead of -- 46105.20

And since I wouldn’t know which part of the quote is being colored a given style, I couldn’t make precise regular expressions that captured it precisely. This is what is known as a “needle-in-a-colored-haystack” kind of problem.

But — not all is lost. I learned a LOT about grabbing things from pages, so I’m sure that skill will come in handy down the line. After realizing that scraping the Tradingview site was a non-starter, I did some digging and found that my charting program I use has data formatted locally on my drive I could parse.

You live and learn, I suppose.

All I can say though is — whoever designed Tradingview’s quote display system is an evil bastard genius.

And I’d buy them a beer.

More to come…

Did I say “fun”? Yes, I did. Though figuring this out initially wasn’t very fun at first. When it comes to price data, you have to know a few things when collating a bunch of historical prices. Namely the duration from one date to the next, or from a point in the past to the present.

My intent with this was to only pull the minimum amount of data from Tradingview, so I didn’t abuse their websocket and cause some admin somewhere to curse at my IP when he viewed the server logs. This requires some time functions, so I set about to make one.

Like any task, sometimes I think “Hey, this is only going to be a few steps.” Then later, when I look up at the clock and its 2am, I realize that I have plumbed a very deep rabbit-hole of specialized knowledge that I only want the barest nuance of.

For instance for my application I want a whole day’s prior data. Turns out there’s 6 bars in a day using a 4-hour period per bar. (6 x 4 = 24, so that is cool.) But, at any given time I start the main polling process, I won’t be precisely 24 hours out, I’ll be 24 + some random interval of time. So I have to calculate the proper offset to get everything up to now.

Okay – no problem, right? Just take a date like March 29th, and subtract one day and start there, right? — well, sure if its the middle or end of the month, what happens when you do it after crossing over to the 1st of the next month?

Oh man…

That means you have to know how many days in a month there are, when they change, and god help you — if its a leap year because Feb will have 29 days instead of 28.

So guess what I did?

I made a function that could determine those transitions – and get this – even CENTURY leap years, which is funny since I’m not going to live long enough to see another one, but hell, I guess I wanted to cover it anyway.

So that is how things sort of expand and become a bit more complex when determining what to do in a program. Here’s what I came up with for basic timestamp stuff:

import datetime
import pytz

timeZone = pytz.timezone('US/Central') # GMT -6 hrs

currDate = datetime.datetime.now(timeZone)

priorNaive = datetime.datetime(2022,3,27,00,00,00)

priorDate = priorNaive.astimezone(timeZone) # Make timezone aware

secElapsed = (currDate-priorDate).total_seconds()

print("Prior Date: " + str(priorDate))
print("Current Date: " + str(currDate))
print("Elapsed seconds: " + str(secElapsed) + "\n")

Which results in two timezone-aware timestamps calculating elapsed seconds:

Prior Date: 2022-03-27 00:00:00-05:00
Current Date: 2022-03-28 21:56:48.813351-05:00
Elapsed seconds: 165408.813351

And just for anyone who needed something that does leap year/century in Python:

import re
import datetime
import pytz
import math

timeZone = pytz.timezone('US/Central') # GMT -5 hrs during CDT, -6 CST

#yearTest = ["2024"]
#dateTest = ["07","04"]

## Month, days - (leap year changes Feb):
## Jan (31), Feb (28 or 29), Mar (31), Apr (30), May (31), Jun (30), Jul (31), Aug (31), Sep (30), Oct (31), Nov (30), Dec (31)
## To be a leap year, the year number must be divisible by 4 except for end-of-century leap years, which must be divisible by 400.
## So the year 2000 was a end-of-century leap year, although 1900 was not. 2024 and 2028 are upcoming leap years
## The below returns YYYY-MM-DD
def dateCalcOffset(myDateList, myYearList): # Checks for leap year/century and month boundary transitions
	months = ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"] # Months for debug, etc..
	numDays = [31,28,31,30,31,30,31,31,30,31,30,31] # Feb can be 29 days if its a leap year
	endOfCentury = "false"
	endOfCentLeap = "false"
	leapYear = "false"
	days = 0
	leapDays = 0
	priorDay = 0
	# Cast these as ints for calcs
	myMonth = int(myDateList[0])
	myDay = int(myDateList[1])
	myYear = int(myYearList[0])
	
	# Leap year - Is it a leap year?
	# First check if its end of century using modulo
	centuryRemainder = myYear % 100
	if centuryRemainder == 0:
		endOfCentury = "true"
		if endOfCentury == "true":
			centuryRemainder = myYear % 400 # If true, check if century leap year
			if centuryRemainder == 0:
				endOfCentLeap = "true"
				#print("Year: " + myYearList[0] + " is an end of century leap year.") # Debug
				# Add extra day if month is February
				if myMonth == 2:
					leapDays = 29
					print("Feb has: " + str(leapDays) + " days.")  # Debug
			else:
				#print("Year: " + myYearList[0] + " is not a end of century leap year.") # Debug
				days = numDays[myMonth-1] # or just return days per month index
				#print(months[myMonth-1] + " has : " + str(days) + " days.") # Debug
	else:
		#print("Year: " + myYearList[0] + " is not the end of a century.") # Debug
		nothing = 0 # Does nothing, but python wants something here if above is commented out lol
		
	# Now check for leap year - only if century check isn't positive
	if endOfCentury == "false" and endOfCentLeap == "false":
		leapRemainder = myYear % 4
		if leapRemainder == 0: # No remainder? Its a leap year
			#print("Year: " + myYearList[0] + " is a leap year.") # Debug
			leapYear = "true"
			# Add extra day if month is February
			if myMonth == 2:
				leapDays = 29
				#print("Feb has: " + str(leapDays) + " days.")  # Debug
			else:
				days = numDays[myMonth-1] # or just return days per month index
				#print(months[myMonth-1] + " has: " + str(days) + " days.") # Debug
		else:
			#print("Year: " + myYearList[0] + " is not a leap year.") # Debug
			days = numDays[myMonth-1] # or just return days per month index
			#print(months[myMonth-1] + " has: " + str(days) + " days.") # Debug
	# Input check - if day is greater than number of days for that month - throw error
	if myDay > days: # Shouldn't happen, but catch it in case...
		print("Input day greater than maximum allowed for month/leap calcs.")
		return
	else: # Do calculations accounting for being first of month from prior month
		if myDay == 1 and endOfCentLeap == "true" or leapYear == "true": # Is it leap year/century?
			if myMonth == 2: # February 1st?
				myMonth = myMonth-1 # Decrement month
				priorDay = numDays[myMonth-1] # Get last day of prior month
			else: # Not the 1st of the month, and not Feb
				if myDay > 1 and myDay <= numDays[myMonth-1]:
					priorDay = myDay-1
		else: # Not a leap year/century
			if myDay == 1: # First of month?
				myMonth = myMonth-1 # Decrement month
				priorDay = numDays[myMonth-1] # Get last day of prior month
			else:
				if myDay > 1 and myDay <= numDays[myMonth-1]: # Not first of month - first week or later?
					priorDay = myDay-1
		
	return myYear, myMonth, priorDay

def getCurrentStamp(myTimeZone): # Takes pytz assigned timezone for calcs
	myCurrentStamp = datetime.datetime.now(myTimeZone)
	
	return myCurrentStamp

def getOffsetStamp(myCurrent, myTimeZone):
	dateRegex = '-(\d\d)' # Splits out stuff after '-' character
	yearRegex = '^(\d\d\d\d)' # Splits out year
	myDateParsed = re.findall(dateRegex, str(myCurrent)) # returns list, element[0] is month, element[1] is day
	myYearParsed = re.findall(yearRegex, str(myCurrent)) # returns list, element[0] is year
	offsetDate = dateCalcOffset(myDateParsed, myYearParsed)
	myPriorNaive = datetime.datetime(offsetDate[0],offsetDate[1],offsetDate[2],00,00,00) ## Year,Month,Day,Hr,Min,Sec
	myPriorStamp = myPriorNaive.astimezone(myTimeZone) # Make timezone aware
	
	return myPriorStamp

def elapsedSeconds(myCurrent, myPrior): # Gets elapsed from two time-zone aware datestamps
	ttlSeconds = (myCurrent-myPrior).total_seconds()
	return ttlSeconds

def numOfBars(mySeconds, barInterval): # Takes total seconds, Bar interval in minutes
	barSeconds = barInterval * 60
	ttlBarNum = math.trunc(mySeconds / barSeconds) # Drop decimals, can't get a fraction of a bar
	
	return ttlBarNum

#--------------- Testing Functions

#today = getCurrentStamp(timeZone)

#offset = getOffsetStamp(today, timeZone)

#secondsBetween = elapsedSeconds(today, offset)

#barResolution = 240 # Minutes

#howManyBars =  numOfBars(secondsBetween, barResolution)

#print("Todays timestamp: " + str(today))
#print("Prior timestamp: " + str(offset))
#print("Elapsed seconds: " + str(secondsBetween))
#print("Elapsed minutes: " + str(math.trunc(secondsBetween/60)))
#print("Elapsed hours: " + str(math.trunc((secondsBetween/60)/60)))
#print("Retrieve " + str(howManyBars) + " bars of historical data.")

Yes, I included the other functions and my commented-out testing statements for completeness. The output looks like this:

Todays timestamp: 2022-03-29 13:09:16.194627-05:00
Prior timestamp: 2022-03-28 00:00:00-05:00
Elapsed seconds: 133756.194627
Elapsed minutes: 2229
Elapsed hours: 37
Retrieve 9 bars of historical data.

Now I’m set to do all kinds of time things with bar data! It only took plenty of TIME to figure it out, lol.

More to come…

Like most things, if I think its going to be easy I usually find some stones I need to hop over to make some progress. Last time, we left off with me using Tradingview’s websockets to grab some quote data. The result of which looks a bit like this:

{"i":0,"v":[1648224000.0,44345.51290507,44590.0,44050.0,44452.0,541.3003276299921]},
{"i":1,"v":[1648238400.0,44452.74643167,44636.0,44275.0,44337.0,318.5197989300029]},
{"i":2,"v":[1648252800.0,44336.0,44477.0,44112.0,44441.0,193.30515490000215]},
{"i":3,"v":[1648267200.0,44434.07106376,44559.28387061,44379.0,44534.02163765,236.48303291000155]},
{"i":4,"v":[1648281600.0,44521.0,44598.0,44329.0,44335.0,158.88092032999873]},
{"i":5,"v":[1648296000.0,44335.0,44406.0,44165.0,44241.0,146.44034252999973]}],
"ns":{"d":"","indexes":[]},"t":"s1","lbs":{"bar_close_time":1648310400}}},

What the eff does that stuff mean? Let me elaborate.

The first line : 1648224000.0,44345.51290507,44590.0,44050.0,44452.0,541.3003276299921 — Unix timestamp, Open, High, Low, Close, Volume
So that timestamp would be – Friday March 25th 2022 11am DST. There’s a lot of numbers after the decimal, I only need two — so that entry would really read:
Open: 44345.51, High: 44590.00, Low: 44050.00, Close: 44452.00 Volume: 541.30
I was able to specify 240 minute (4-Hour) bars which makes it pretty easy to get a whole day in just a few entries, since 6 bars = 24 hours. Seems easy, right? All I need to do now is to parse that and write it in a way that makes sense for my encoder.

Here it is looking a bit more formatted:

Index,Date,Open,High,Low,Close,Volume
0,"03/25/2022, 03:00:00",43911.22112839,44654.67307908,43606.0,44600.0,915.298048519977
1,"03/25/2022, 07:00:00",44601.0,45082.0,44236.0038521,44346.0,1818.5557992299166
2,"03/25/2022, 11:00:00",44345.51290507,44590.0,44050.0,44452.0,541.3003276299921
3,"03/25/2022, 15:00:00",44452.74643167,44636.0,44275.0,44337.0,318.5197989300029
4,"03/25/2022, 19:00:00",44336.0,44477.0,44112.0,44441.0,193.30515490000215
5,"03/25/2022, 23:00:00",44434.07106376,44559.28387061,44379.0,44534.02163765,236.48321299000156
6,"03/26/2022, 03:00:00",44521.0,44598.0,44329.0,44335.0,158.88092032999873
7,"03/26/2022, 07:00:00",44335.0,44406.0,44165.0,44211.0,196.8329842700001
8,"03/26/2022, 11:00:00",44207.0,44472.0,44152.89142732,44367.0,138.23807569000022
9,"03/26/2022, 15:00:00",44364.0,44785.0,44257.0,44473.0,372.51797744998487

Next steps will be getting more than one quote at a time, which should be possible. In the end I’ll have quite a few of them interleaved among each other, which means I need to lean hard on regular expressions to sift through the sea of data.

(Some time later)

I’m deep into Regular Expressions, a way to sift through the alphabet soup of data and pick out the things that I want. There’s some nice tools out there to help, like regex101 dot com, but its still pretty arcane syntax-wise.
Making some progress, but its getting tricky. Let me explain. I’m using websockets, so I see the data coming from the server and it gets dumped to the console. Problem is, using Regex means it parses whatever it gets its little grubby hands on, which means it could be influenced by debug messages I dump to the console too – a bit like double-dipping into a stream.
So I have to figure out how to debug the program without messing up the datasource. Or at least I think I do at this point. Its messing with my head 🙂
I might be able to get ahead of it by flagging my debugging messages in a way so that it will ignore that, but work on the other data. Maybe… or… split out the results and save them to a file so it doesn’t “pollute” the same stream of data I’m trying to parse.

(Which is really what I should be doing, I think.)

Hoo boy, my head hurts. But I think I have it finally.

This is the data that I’ve been dealing with — just so you have an idea what it looks like raw from the websocket itself:

quote_session ID generated qs_yngjrgxzshkg
chart_session ID generated cs_vfiuozaqmkwh
~m~361~m~{"session_id":"<0.18544.193>_sfo-charts-18-webchart-5@sfo-compute-18_x","timestamp":1648480416,"timestampMs":1648480416379,"release":"registry.xtools.tv/tvbs_release/webchart:release_205-53","studies_metadata_hash":"79c6b847bdfc53283f5b5f6e28f71f7baa91e9f2","protocol":"json","javastudies":"javastudies-3.61_2183","auth_scheme_vsn":2}

~m~484~m~{"m":"qsd","p":["qs_yngjrgxzshkg",{"n":"BITFINEX:BTCUSD","s":"ok","v":{"volume":4537.63954264,"update_mode":"streaming","type":"crypto","short_name":"BTCUSD","rtc":null,"rchp":null,"pro_name":"BITFINEX:BTCUSD","pricescale":10,"original_name":"BITFINEX:BTCUSD","minmove2":0,"minmov":1,"lp_time":1648480412,"lp":47748.0,"is_tradable":true,"fractional":false,"exchange":"BITFINEX","description":"Bitcoin / Dollar","current_session":"market","currency_code":"USD","chp":2.0,"ch":935.0}}]}~m~65~m~{"m":"quote_completed","p":["qs_yngjrgxzshkg","BITFINEX:BTCUSD"]}

So after many attempts that failed, I finally came up with some regex that could filter it into this:

BITFINEX:BTCUSD
Volume: 4537.63979476
Price: 47748.46935417
BITFINEX:BTCUSD
Volume: 4537.67328922
Price: 47748.0
BITFINEX:BTCUSD
Volume: 4537.94068011
Price: 47734.0
BITFINEX:BTCUSD
Volume: 4538.05830511
Price: 47737.0
BITFINEX:BTCUSD
Volume: 4538.90654622
Price: 47732.0
BITFINEX:BTCUSD
Volume: 4539.06445621
Price: 47727.62184819

It took quite a bit to get that all working. Here’s a sample of some of the regex I used:

priceRegex = '\"lp\":(\d+.\d+)'

Make sense to you? Me either, which is why I’m super-glad that sites like regex101 dot com exist. Next, I’ll have to figure out duration between two dates in order to calculate how much quote data to ask for historically.

Until next time…