Category: Modding

OC – Dev Sketches

Modding

It takes time to make a mod in Teardown. Sure, you can just belt something out and push it on to the workshop, but the mods that seem to endure are the ones that had more than just 15 minutes spent on their design.

I’m a visual person, so when I’m working I like to sketch out things to figure out a specific problem, or just to have a visual representation of a bit of code that I’m working on. It keeps me on track, and is a valuable tool to double-check that I’m attacking something properly to solve it.

Here’s a few of the drawings I made while making the Omnicar, with some brief explanation of what is going on.

Remember Trigonometry? I sure didn’t, so I had to get some brush-up courses and relearn a few things. (I picked up a good deal via Humble Bundle for Gamedev dot tv “Math for games” courses. Cost me $5 bucks, lol.) One of my problems was locking a plane of coordinates to an arbitrary moving target, which required a few calculations. This diagram is basic Trig, properties of Sine/Cosine and how they relate to a unit circle.

The degree markings are how they are represented in-game. In Teardown you get 180 degree plus/minus, switching over at the 0 mark. The quadrants just represent which is positive at that degree of the circle, which helps with troubleshooting things.

Oh hypotenuse, we hardly knew thee! Seriously though, this came out to be more useful than I originally anticipated. Figuring out angles using right-triangles is quite easy and Hypotenuse lengths are useful if you’re doing things around an origin point.

The heavy-lifting of calculating tangent coordinates. I wanted to put sprites on a row of these, so they’d stay with the car and be properly perpendicular. I had to do some other steps that I don’t outline here, but this diagram helped me quite a bit to get it going.

Do you like dynamically resizing sprites? I sure do. This was the hairy bit that I had to figure out with the existing tangent coordinate calcs. Bunch of values that I was working with on a test case, just to make sure that I had it all nailed down before feeding the Sprite Renderer coords and values.

I went through quite a few iterations on how to make the “skateboard” design drive more consistently. Initially the rear of the car would pivot and overtake the front, spinning it around at times. It was very “loose” in its characteristics, so I had to design something that would dampen the Y-Sway. I’m happy to say that I came up with something that worked out.

Using any of the API functions that contain the “Constrain” keyword can yield some rather hilarious results. At one point, I tried a test where it immediately turned the car into a whirlwind of broken parts ascending into the sky. That was quite funny, now that I look back on it.

Figuring out acceleration curves. From what I can tell, most vehicles in Teardown use a linear acceleration model. That works for most things, but I wanted something with a bit more finesse. Enter Quad Bezier curves! With a start, mid, end point I could make a curve to my liking.

The real trick was doing the rescaling needed to take arbitrary bezier 2D coordinates and make them relate to the speed I was adding to my vector forces. That, and making a timer mechanism that allowed me to control the duration of the curve itself, feeding the Bezier function “live” so it would return a proportional speed value in relation to the maximum speed limit.

Did I tell you it got complicated? Oh yeah, it sure as hell did. But it works, so that is what matters. Its pretty nice to be able to “nudge” the forward control and get a smooth rise in speed – so you can control things on the low-end but also do a full burnout to maximum.

Did I mention the bezier curve is adjustable? Oh yes, it is. I haven’t put a visualization of it in the UI, but its in there.

There’s more to come as I complete things. I think I’m at the 90% stage in the project, which means I’m getting impatient to finish and I want to work on something else. I’ll get there though and release it, I promise.

Until next time…

OC – The “Skateboard”

Modding

Where I left off in the last post, I mentioned that I had to use “game logic” to figure out a good way to get the ball wheel design working. Funny thing, I did exactly that. In the real world, we don’t have things that can pass through other objects like a ghost. It might be handy, but that kind of thing isn’t part of our reality.

After pondering the roller/cage problem for a while, I realized — I was thinking in too literal a fashion. I didn’t have to make a containment system like I would if this was a physical object. The solution was to have a support strut that went into the center of the sphere wheel with a ball joint at the end.

The result looked like this:

Why red support struts? Just for visibility so I could see what was going on…

In the background there you can see the blue crane-like rig I used to figure out if the wheels would work at all. It was fun putting it together and making it move around. I call this design the “Skateboard” because there is absolutely no way to directly power the wheels. I was stumped for a bit until a fellow modder Dima helped me out.

The answer? I was trying to apply Body Impulses using the modding API to the frame, but the results were not what I had hoped for. Dima showed me that applying vectors of force to the pivot was the way to go, in addition to using Constraints so the speed didn’t get out of hand.

With that under my belt, I was able to hack something together that not only used a “front drive” but something I like to call “crab drive” — where force is applied to the center of mass so all the wheels turn in the same direction. Think parallel parking and you want to go sideways into a spot. Well if this was on your car, you would have no trouble at all.

Demonstrating front drive mode, then crab mode. Watch the difference in steering.

Not bad, but I’m sure you see the problem in front drive mode — the rear wheels tend to just roll with momentum so sometimes they start to overtake the front and rotate the entire body. That isn’t something I wanted. Crab mode however is rock solid because center of mass is a pretty stable place to apply forces from.

This was my first problem – how to use front drive mode without the back end swishing everywhere? Some might prefer the out-of-control feel of driving that way, but I knew most would be expecting conventional drive, where small adjustments to steering didn’t cause the back end to shimmy all over.

After some research I realized that I needed to apply some counter-force to the rear of where the front drive forces were being applied. In Teardown, the pivot point of the body determines where those forces get placed, unless you calculate the center of mass like I had.

So this meant I needed to figure out how to create a virtual pivot point to apply forces so the tendency to turn/swivel after each forward push was minimized. Let me tell you, that wasn’t too easy. I never had higher math courses and I’ve only recently dove in to Trigonometry, like this diagram below from Gamedev dot tv:

Is your brain melting yet? Oh trust me, it will…

That diagram shows what is needed to solve an arbitrary angle using math for a right triangle. Yep, games have lots of math in them, but its all hidden away in functions and other bits.

I had to figure out somehow this “Virtual Pivot” problem before I could even hope to stabilize the driving behavior of Omnicar.

(To be continued)

OC – Prototyping

Modding

I get ideas from all kinds of sources. Sometimes, I see something neat in a movie, or I look at a website and there’s a cool concept that spurs another idea in my head. Well, I was puttering around and found this video via Hackaday of a lego-ball-driven car.

Bright sphere wheels and some lego motors, what is there not to love?

I have a project list, so I can jot down an idea and assign it a number. I don’t do these sequentially, typically I follow the classic pattern of Teardown (or any other game) modding:

I do actually finish some of them. I do like to rotate between different projects when I feel I’ve hit a wall, or I just am sick of looking at the same thing for months on end. That break usually allows for more thinking about how to improve things in general.

This particular idea was number 25. (As of this writing I have about 47 ideas and I add to that on an ongoing basis.) The concept was interesting to me for a few reasons:

  1. There is a built-in vehicle system/entity in Teardown, but its a bit finicky and it is very easy to crash the editor if you don’t get everything just right.
  2. The vehicle entity requires a setup that is Magica-native, in terms of a compound model with shapes nested in its structure. (Yet another thing I really don’t want to mess with. I like Qubicle anyway.)
  3. Vehicles have some limitations when it comes to behavior of lights – they only turn on when you’re driving, for instance. There are some hacky ways around it, but I didn’t like any of those approaches.
  4. Driving using regular wheels means accepting the suspension/turning design, it is very hard to change that substantially with the vehicle entity.
  5. Omnisphere wheels are cool as hell, and enable neat tricks like “Crab Mode” when driving.
  6. I could make my own light systems and control them however I wanted, including everything else, like chase cams, etc…

So omnicar it is!! My first attempt was rather — plain. But you have to start somewhere, right?

The vertical bits were to keep the spheres contained, but I figured out a better way later on.

For all intents and purposes, this was the base chassis of the car. Those wheels? I know they look a bit clunky, but the Teardown engine treats individual voxel cubes as spheres for collisions, so I knew they would roll okay. I just had to figure out how to get that done.

This involved making a in-game test rig, where I messed with joints (an entity type that does what it says, links two parts together with varying degrees of freedom) to find the optimum solution. My first try was a cage-like system lined with rollers, but that didn’t perform too well.

The collisions would slow things down and generally it was really clunky. However, I did come up with something — I just had to apply “game logic” to my design process.

(To be continued…)

Jitter And Raycast

Modding
Just messing with spawning single voxels – and debugging coordinates.

So, been working on different things in this game engine, and one thing I always use is the X, Y, Z coordinates of an object.


This game has things defined in meters/centimeters, but in any game wherever something exists, it has a coordinate to place it in 3D space. When I first started, I ran across a problem with the numbers, namely they updated all the time because the game engine was simulating physics, so it introduced variations in the numbers out to nth-decimal places.


The raw data looked something like this:

Watch those numbers go!

When you play that, notice how Y seems to move no matter what, changing values constantly. The others do it too, but not as much.

The reason for this is that “Y” in the game is the axis that is perpendicular to the ground, so physics is always trying to figure out where that coordinate is for an object since there’s simulated gravity. At the same time, the ground plane is a colliding object, so it prevents the things on top of it from falling.

This results in a bunch of changing values, or “Jitter”. As you can see, the number of decimal places is staggering — 14 or so places! I don’t need that kind of precision for most things. In fact, I can get away with something like one or two decimal places.

Problem is, even if you round to the nearest decimal, one or two places, the numbers can change enough that you still get “Jitter”, even at that reduced resolution.

So what to do? Well, fortunately others have tackled this problem and I ran into a solution in a gaming forum for guys using custom controllers as inputs to flying simulators.

Here’s the code I came up with:

It looks more complicated than it is, because I’m doing it for all three axis — X, Y, Z at the same time.

The basic jist is, if I get a Jittery X input greater than an older stored X value OR its less than the stored value minus a threshold — then I store it and pass it on. If it fails any of those tests, I just keep the prior old stored value and don’t update it.

This has the effect of throwing out short-term fluctuations and results in a much smoother experience. (It essentially creates a band of values from oldX – threshold to oldX and chucks out any change that falls within it.) The variable is defined outside the function as —

threshold = 0.15 -- Jitter threshold

…and it gets referenced once we’re in there.

As I was working on the raycast part of my code, I noticed in-game I was getting some odd results. I was doing some spawning of objects, which looks like this:

Spawn("MOD/vox/SomeVoxModel.vox", Transform(Position, Rotation), true)

It just looks up something and spawns it in there. Easy-peasy, right? Sure.

I wanted to make something where you could spawn objects in an accurate way, which is where I ran into a problem using voxboxes. (A voxbox can have properties set in code, versus loading a static vox model.)

To do some of the things I wanted, I needed to be able to detect where the player was looking, so I knew where to put the next spawned object. This was easy enough using a function:

-- QueryRaycast: Origin, Direction, Maxdist, Radius, RejectTransparent true/false
    rayHit, shapeDistance, worldNormalList, shapeHandle = QueryRaycast(GetCameraTransform().pos, Direction, 10, 0, false)

It takes a bunch of arguments and then returns a bunch of data that gets stored in the variable names I have on the left side of the equals sign.

If it hit something, how far, the face “normal” which is just the direction of the face on an object, and the handle – which is a number that the game uses to track things. Each object gets one.

So I was humming along and things were working, but I needed to make sure that the “ray” would report the proper coordinates when needed. So I tested it using a voxbox. As I mentioned earlier, Its different than the regular models in the sense that you can change a lot of its properties, like color or material in code – which makes it more flexible than just importing a static model.

Determined to find the problem, I whipped up a test, but then things went wrong.

Here’s a video to show what I mean:

Seems I can only get location on one corner of the voxbox, which is wrong!

What you see there is me aiming at the object, but I only get data if I’m aiming at a specific corner of it. That is really weird/wrong, so I tested it with a shape that I loaded — which is different than the voxbox — and it worked for that one.

This made zero sense – it should work equally well for both.

So then I made ANOTHER test, this time documenting it in detail and making it as simple as possible. This time I used a function called:

GetShapeBounds(shapeHandle)

This takes an object ‘handle’ number and returns two coordinates — one for the uppermost corner, and one for the lowermost corner. I put some debug cross things in there so I could see it visually when the game was running, and what I found out was — the cube had a ‘ghost’ that was connected at the lower ‘point’ of the cube and offset diagonally from it.

This shouldn’t be the case.

So I zipped up the demo and code, and posted it where the developer would see it.

I’m happy to say that I got a response – and they said “Yep, something’s wrong, gotta ask the guy that made the engine what is happening.” So, I found a bug and he’s gonna try to fix it. I’m sure he loves me right now, lol.

I did get a response not long ago and he indicated that it should be a simple fix. Which is awesome, and I got the nerd cred for finding an original bug in the engine itself!

Mechanic – Proof Of Concept

Modding

Ever hear of the term “Nerd Sniping”? I hadn’t until a bit ago, but this totally happened to me. No, it isn’t a bad thing. It refers to a situation where someone says “…that of course, is impossible” (or merely — “how can I do this”) and someone with an engineering/programming mindset thinks, “wait a minute, I could solve that”.

So one day I’m hanging out talking with some fellow modders about this and that, and the topic comes up about a certain game mechanic, (differentiating here between game vs real-life auto mechanics, lol) and how to approach a given problem.

The more we chatted, I realized my brain was already working toward the solution. The code was writing itself in my head. I have a pile of projects that I’m going to get to – and this game mechanic of being a virtual mechanic fixing/repairing things is on that list.

I dove in, thinking it would be super-easy. (Turned out to be a bit more involved, of course.)

Here’s an early alpha where I was figuring out some basics, like highlighting the proper part and deciding how to make it function.

Testing that my raycast registers the proper part and the highlight works for valid parts.

A hundred plus lines of lua later I had the basic idea done for removing a part and “storing” it somewhere. The idea was to have things you’d disassemble or repair, with visual prompts to help the player along.

The “ghost” placeholder was the same vox model, just set to not collide and a bit of glow.

All well and good, but it wasn’t complete. The part in the players hands had to end up where the visual indicator was, just to make it work right. This took a bit more work, where I was storing different states in arrays (though now I think of it, structs would be better — I’ll fix it later) while the player manipulated and placed things.

Its funny, when I work on things sometimes I hit some problem and either a) work around it at the time, or b) get annoyed and drill down to the real reason its behaving that way.

Well, this time it involved coordinate systems. If you recall cartesian coordinates from high school, you have your x and y axis, that can define a point with two coordinates, like (5, 3).

I’m working in three dimensions, so I have X, Y, Z.

In Teardown, the Y axis always points upward. Its just how its built. My problem is when I do the sleight-of-hand needed to ‘snap’ the part into place (I delete the ghost shape and spawn the original part where it should be.) The shape doesn’t end up where it should.

Its CLOSE, but offset enough that it is causing problems.

So I debugged by printing all these variables to the console to see what the hell was going on, but I wasn’t sure exactly what it was yet. I needed the model to spawn at say: -3.4, 0.4, 0.2 (X, Y, Z) and it wasn’t doing that exactly. Its like trying to add 2 + 2 but getting 5.165 instead of 4.

My solution was to construct a careful test so I could drill down on why its doing this. There are some possibilities where errors would creep in — world vs local coordinates, the shapes and bodies encapsulating them.

I had to find the offset that was creeping in somehow and stomp that bug flat.

Well, wouldn’t you know — I found out why. Here’s the final proof-of-concept-demo:

Part can be selected, placed and then reinstalled if need be. Note the extra states in the upper-left.

It was the vox model itself. (I thought it was in the code possibly, which was driving me nuts.)

Each model has what is called a ‘pivot’, which is just a axis that can be adjusted for various reasons. The model I was using came from elsewhere (and critically I didn’t check beforehand, my mistake), and turns out the pivots were slightly off — about 0.5 on every axis. The physical location of the model in world coordinates was a bit offset too.

Bit of a hair-puller, but it taught me a lesson — check your assets! I really should’ve looked at it more closely before working with it, since it was part of a compound model and those offsets were there from that particular type of assembly.

Another lesson was attempting to map states for an object.

Usually you can get away with setting some boolean (true/false) flags and call it a day, but this particular mechanic required no less than NINE different states when I broke it down. No wonder I was having trouble earlier! I can only keep so many things in my head at once, lol.

Diagramming and pseudo-code breakdowns were a big help. Sometimes you get so lost in the weeds of a problem that you need to make a map of the destination to keep you on track.

I’m always learning something — and this dovetails nicely into a future post how I found out a quirk of the engine… but more on that later.

(Cheers to Nathan who inspired me to figure this out – here’s a link to his project site.)

VTF – Polling And Tick Data

Modding

Last time I left off at ditching Tradingview as a quote source. These things happen, and I learned quite a lot along the way, so I’m not troubled by it. Turning locally, as in my own hard drive, I began sifting through the data that gets archived by my chart provider.

Fortunately they aren’t super-protective of their quotes, so the files were essentially text data with some custom file extensions on the end. I began Python-ing a way to get these things filtered and saved in the proper folder for my Teardown mod.

The data is quite orderly:

Date, Time, Open, High, Low, Last, Volume, # of Trades, OHLC Avg, HLC Avg, HL Avg, Bid Volume, Ask Volume
2022/4/4, 13:22:45.0, 4561.50, 4561.50, 4561.50, 4561.50, 3, 3, 4561.50, 4561.50, 4561.50, 0, 3

After a bit of filtering/detection, I had it doing things like this:

Last line in file: ESM22-CME-BarData.txt is:
2022/4/4, 20:32:40.0, 4571.25, 4571.25, 4571.25, 4571.25, 3, 3, 4571.25, 4571.25, 4571.25, 3, 0

Symbol: ES Last price: 4571.25 Volume: 3
File size in bytes: 12587

(I’m counting filesize so I can wipe it when it hits a certain size, to keep search times down for some functions.)

One main task I had to figure out was filtering based on futures symbol. Most symbols follow the convention of:

 <symbol><monthcode><YY>

As with anything, there are a few exceptions where the year/month is flipped, but in aggregate my initial function seemed to do the trick.

First try, and boy did I go into deep IF … land on this one.

I realized later on that this first pass wasn’t good enough – it wasn’t behaving EXACTLY as I needed, so it made sense to go through and refactor it into a function (or series of them), that worked how I wanted. Sometimes being “cute” and trying to do too many things at once creates technical debt that bites you in the ass when you least expect it.

After some pseudo-code restructuring to actually plan the flow of what I was trying to do, I came up with this result:

Sometimes breaking up things makes it much cleaner and easier to debug.

I set up a test where I threw it every symbol I had in my local directory to make sure it could handle things:

Remember, I just wanted futures symbols. Stocks and non-recognized/allowed ones should “fail”.

So far, so good! It was nice being able to chuck anything at it, even the dollar-sign prefix ones, and have it handle things in a sane way.

I now had to plan out the pseudo-code for making the main polling loop work. I’m not a Python expert by any means, so I just worked from some examples and adapted it to my purpose. After thinking a bit, I managed to get this flow:

The glorious refactoring pass of my data flow. This document changes as I go along, but its a good way to outline the major steps in execution.

This happens to me sometimes, I’m humming along and Python-ing my way to a given goal, and something gets thrown in my way that I didn’t quite expect.

Debugging isn’t glamorous, there aren’t any hacker-esque (in the hollywood sense) terminals with interesting things being scrolled/displayed/animated on them. Its just my boring Notepad++ open to a file and the shell where I’m running Python commands.

At one point, things were silently failing.

It could be my fault – I’m not an expert, as I said – so I could be structuring things too deeply in terms of logic IF .. THEN or just calling a bunch of functions like a fool. The result was I had to place some Debug statements like — print(“Did I even get to this point?”) in the code to figure out WHERE it was dying.

Not fun, in the least.

At this point, I’m nearly at the end of the Python part of implementing this mod — the rest of the steps are to get the voxel model encoded and saved to the proper directory, then in Teardown have it get read/decoded by a lua script.

So very close…. more to come.

VTF – Quotes And Clever Bastards

Modding

There I was, puttering right along and getting some quote data when the unthinkable happened — Tradingview got wise to my quote-scraping ways, and the historical data I successfully was getting turned into error messages from the server. Dang.

The good ‘ol days, when quotes worked and historical data grew on metaphorical trees.

Oh, cruel fate.

I had to regroup and try to salvage something. I had a bunch of regular expressions for filtering Tradingview’s price data, and I didn’t want them to go to waste! After a bit of thinking, I decided to do it the old fashioned way — scrape the Tradingview site directly.

This would require learning yet-another-skill, using a module called “Selenium” for Python. This clever library allows you to dive into page source code for stuff, or open up web pages and even simulate user “clicks” and data entry – for say, logging in.

I got to work, and soon I had something going:

browser.get(url)
browser.implicitly_wait(3)
## Have to use CSS selector when class names have spaces - replace with '.'
login_button_class = "tv-header__user-menu-button"
user_menu_class = "item-4TFSfyGO"
sign_in_email_class = "tv-signin-dialog__toggle-email"
user_name = "username"
browser.find_element(by=By.CLASS_NAME, value=login_button_class).click() # Click on user login
time.sleep(1)
browser.find_element(by=By.CLASS_NAME, value=user_menu_class).click() # Click on dropdown for email
time.sleep(1)
browser.find_element(by=By.CLASS_NAME, value=sign_in_email_class).click() # Click on email for User/Pass dialog
time.sleep(1)
input_username = browser.find_element(by=By.NAME, value=user_name).click() # Click on email for User/Pass dialog
pyautogui.typewrite(user) # works
pyautogui.press("tab") # Tab to next field
pyautogui.typewrite(password)
pyautogui.press("enter")
time.sleep(1)

There’s more setup involved prior to these actions, but I wanted to show the core of what I was doing. Clicking buttons, signing in, all of that. There was a way to cache the result — so I didn’t have to log in every single time, but for some reason that code didn’t work for me. I plowed ahead, undaunted.

A few bazillion log-ins later, and a bunch of “did you just log in from a new device” emails, I was at the page where you could add symbols to a watchlist, and it would helpfully display them on the right hand side of the page. I was set! (Or so I thought.)

No my friends, no such luck. Turns out the Tradingview chaps are quite resourceful. Let me explain.

In the “old days” you could look at a website’s source, it would have its data embedded in the page like “lastprice=46624.50”, which was trivial to scrape.


Well, websites are now reactive and do all kinds of things, which means what I was searching for was deep in the source. And I mean DEEP. Take a look at this relative path here:

/html/body/div[1]/div/div[1]/div[1]/div[1]/div[1]/div[2]/div/div[2]/div/div/div[2]/div/div[4]/div/div/span[1]/span

And that is just for ONE quote, mind you. (20-plus levels deep!)

Even if you got down there, Tradingview made sure to make it as hard as possible. How? Well, if you weren’t paying much attention, you’d pull up your watchlist and it would have some symbols with prices, like this:

DXY 98.62 BTCUSD 46639.25

So just dive down into the source and get it, right? Well, its more complicated than that. They don’t just display the prices in one go — oh no — some evil genius over there decided on any up/down tick to color a RANDOM portion of the quote green or red.

Which means a simple quote of:

46105.20

Turns into:

<span class="inner-ghhqKDrt">4610<span class="minus-ghhqKDrt">5.2</span></span>

So what, right?

It turns out that its monumentally harder to scrape a quote when the style of that quote changes on a whim. So, part of it is white, some of it is red/green at any given point. By splitting the quote apart in a random way, it turns out regular expressions that you’d use to grab it only get a fraction of the “normal” part:

4610 -- instead of -- 46105.20

And since I wouldn’t know which part of the quote is being colored a given style, I couldn’t make precise regular expressions that captured it precisely. This is what is known as a “needle-in-a-colored-haystack” kind of problem.

But — not all is lost. I learned a LOT about grabbing things from pages, so I’m sure that skill will come in handy down the line. After realizing that scraping the Tradingview site was a non-starter, I did some digging and found that my charting program I use has data formatted locally on my drive I could parse.

You live and learn, I suppose.

All I can say though is — whoever designed Tradingview’s quote display system is an evil bastard genius.

And I’d buy them a beer.

More to come…

VTF – Fun With Time

Modding

Did I say “fun”? Yes, I did. Though figuring this out initially wasn’t very fun at first. When it comes to price data, you have to know a few things when collating a bunch of historical prices. Namely the duration from one date to the next, or from a point in the past to the present.

My intent with this was to only pull the minimum amount of data from Tradingview, so I didn’t abuse their websocket and cause some admin somewhere to curse at my IP when he viewed the server logs. This requires some time functions, so I set about to make one.

Like any task, sometimes I think “Hey, this is only going to be a few steps.” Then later, when I look up at the clock and its 2am, I realize that I have plumbed a very deep rabbit-hole of specialized knowledge that I only want the barest nuance of.

For instance for my application I want a whole day’s prior data. Turns out there’s 6 bars in a day using a 4-hour period per bar. (6 x 4 = 24, so that is cool.) But, at any given time I start the main polling process, I won’t be precisely 24 hours out, I’ll be 24 + some random interval of time. So I have to calculate the proper offset to get everything up to now.

Okay – no problem, right? Just take a date like March 29th, and subtract one day and start there, right? — well, sure if its the middle or end of the month, what happens when you do it after crossing over to the 1st of the next month?

Oh man…

That means you have to know how many days in a month there are, when they change, and god help you — if its a leap year because Feb will have 29 days instead of 28.

So guess what I did?

I made a function that could determine those transitions – and get this – even CENTURY leap years, which is funny since I’m not going to live long enough to see another one, but hell, I guess I wanted to cover it anyway.

So that is how things sort of expand and become a bit more complex when determining what to do in a program. Here’s what I came up with for basic timestamp stuff:

import datetime
import pytz

timeZone = pytz.timezone('US/Central') # GMT -6 hrs

currDate = datetime.datetime.now(timeZone)

priorNaive = datetime.datetime(2022,3,27,00,00,00)

priorDate = priorNaive.astimezone(timeZone) # Make timezone aware

secElapsed = (currDate-priorDate).total_seconds()

print("Prior Date: " + str(priorDate))
print("Current Date: " + str(currDate))
print("Elapsed seconds: " + str(secElapsed) + "\n")

Which results in two timezone-aware timestamps calculating elapsed seconds:

Prior Date: 2022-03-27 00:00:00-05:00
Current Date: 2022-03-28 21:56:48.813351-05:00
Elapsed seconds: 165408.813351

And just for anyone who needed something that does leap year/century in Python:

import re
import datetime
import pytz
import math

timeZone = pytz.timezone('US/Central') # GMT -5 hrs during CDT, -6 CST

#yearTest = ["2024"]
#dateTest = ["07","04"]

## Month, days - (leap year changes Feb):
## Jan (31), Feb (28 or 29), Mar (31), Apr (30), May (31), Jun (30), Jul (31), Aug (31), Sep (30), Oct (31), Nov (30), Dec (31)
## To be a leap year, the year number must be divisible by 4 except for end-of-century leap years, which must be divisible by 400.
## So the year 2000 was a end-of-century leap year, although 1900 was not. 2024 and 2028 are upcoming leap years
## The below returns YYYY-MM-DD
def dateCalcOffset(myDateList, myYearList): # Checks for leap year/century and month boundary transitions
	months = ["Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"] # Months for debug, etc..
	numDays = [31,28,31,30,31,30,31,31,30,31,30,31] # Feb can be 29 days if its a leap year
	endOfCentury = "false"
	endOfCentLeap = "false"
	leapYear = "false"
	days = 0
	leapDays = 0
	priorDay = 0
	# Cast these as ints for calcs
	myMonth = int(myDateList[0])
	myDay = int(myDateList[1])
	myYear = int(myYearList[0])
	
	# Leap year - Is it a leap year?
	# First check if its end of century using modulo
	centuryRemainder = myYear % 100
	if centuryRemainder == 0:
		endOfCentury = "true"
		if endOfCentury == "true":
			centuryRemainder = myYear % 400 # If true, check if century leap year
			if centuryRemainder == 0:
				endOfCentLeap = "true"
				#print("Year: " + myYearList[0] + " is an end of century leap year.") # Debug
				# Add extra day if month is February
				if myMonth == 2:
					leapDays = 29
					print("Feb has: " + str(leapDays) + " days.")  # Debug
			else:
				#print("Year: " + myYearList[0] + " is not a end of century leap year.") # Debug
				days = numDays[myMonth-1] # or just return days per month index
				#print(months[myMonth-1] + " has : " + str(days) + " days.") # Debug
	else:
		#print("Year: " + myYearList[0] + " is not the end of a century.") # Debug
		nothing = 0 # Does nothing, but python wants something here if above is commented out lol
		
	# Now check for leap year - only if century check isn't positive
	if endOfCentury == "false" and endOfCentLeap == "false":
		leapRemainder = myYear % 4
		if leapRemainder == 0: # No remainder? Its a leap year
			#print("Year: " + myYearList[0] + " is a leap year.") # Debug
			leapYear = "true"
			# Add extra day if month is February
			if myMonth == 2:
				leapDays = 29
				#print("Feb has: " + str(leapDays) + " days.")  # Debug
			else:
				days = numDays[myMonth-1] # or just return days per month index
				#print(months[myMonth-1] + " has: " + str(days) + " days.") # Debug
		else:
			#print("Year: " + myYearList[0] + " is not a leap year.") # Debug
			days = numDays[myMonth-1] # or just return days per month index
			#print(months[myMonth-1] + " has: " + str(days) + " days.") # Debug
	# Input check - if day is greater than number of days for that month - throw error
	if myDay > days: # Shouldn't happen, but catch it in case...
		print("Input day greater than maximum allowed for month/leap calcs.")
		return
	else: # Do calculations accounting for being first of month from prior month
		if myDay == 1 and endOfCentLeap == "true" or leapYear == "true": # Is it leap year/century?
			if myMonth == 2: # February 1st?
				myMonth = myMonth-1 # Decrement month
				priorDay = numDays[myMonth-1] # Get last day of prior month
			else: # Not the 1st of the month, and not Feb
				if myDay > 1 and myDay <= numDays[myMonth-1]:
					priorDay = myDay-1
		else: # Not a leap year/century
			if myDay == 1: # First of month?
				myMonth = myMonth-1 # Decrement month
				priorDay = numDays[myMonth-1] # Get last day of prior month
			else:
				if myDay > 1 and myDay <= numDays[myMonth-1]: # Not first of month - first week or later?
					priorDay = myDay-1
		
	return myYear, myMonth, priorDay

def getCurrentStamp(myTimeZone): # Takes pytz assigned timezone for calcs
	myCurrentStamp = datetime.datetime.now(myTimeZone)
	
	return myCurrentStamp

def getOffsetStamp(myCurrent, myTimeZone):
	dateRegex = '-(\d\d)' # Splits out stuff after '-' character
	yearRegex = '^(\d\d\d\d)' # Splits out year
	myDateParsed = re.findall(dateRegex, str(myCurrent)) # returns list, element[0] is month, element[1] is day
	myYearParsed = re.findall(yearRegex, str(myCurrent)) # returns list, element[0] is year
	offsetDate = dateCalcOffset(myDateParsed, myYearParsed)
	myPriorNaive = datetime.datetime(offsetDate[0],offsetDate[1],offsetDate[2],00,00,00) ## Year,Month,Day,Hr,Min,Sec
	myPriorStamp = myPriorNaive.astimezone(myTimeZone) # Make timezone aware
	
	return myPriorStamp

def elapsedSeconds(myCurrent, myPrior): # Gets elapsed from two time-zone aware datestamps
	ttlSeconds = (myCurrent-myPrior).total_seconds()
	return ttlSeconds

def numOfBars(mySeconds, barInterval): # Takes total seconds, Bar interval in minutes
	barSeconds = barInterval * 60
	ttlBarNum = math.trunc(mySeconds / barSeconds) # Drop decimals, can't get a fraction of a bar
	
	return ttlBarNum

#--------------- Testing Functions

#today = getCurrentStamp(timeZone)

#offset = getOffsetStamp(today, timeZone)

#secondsBetween = elapsedSeconds(today, offset)

#barResolution = 240 # Minutes

#howManyBars =  numOfBars(secondsBetween, barResolution)

#print("Todays timestamp: " + str(today))
#print("Prior timestamp: " + str(offset))
#print("Elapsed seconds: " + str(secondsBetween))
#print("Elapsed minutes: " + str(math.trunc(secondsBetween/60)))
#print("Elapsed hours: " + str(math.trunc((secondsBetween/60)/60)))
#print("Retrieve " + str(howManyBars) + " bars of historical data.")

Yes, I included the other functions and my commented-out testing statements for completeness. The output looks like this:

Todays timestamp: 2022-03-29 13:09:16.194627-05:00
Prior timestamp: 2022-03-28 00:00:00-05:00
Elapsed seconds: 133756.194627
Elapsed minutes: 2229
Elapsed hours: 37
Retrieve 9 bars of historical data.

Now I’m set to do all kinds of time things with bar data! It only took plenty of TIME to figure it out, lol.

More to come…

VTF – Parsing The Sea Of Quotes

Modding

Like most things, if I think its going to be easy I usually find some stones I need to hop over to make some progress. Last time, we left off with me using Tradingview’s websockets to grab some quote data. The result of which looks a bit like this:

{"i":0,"v":[1648224000.0,44345.51290507,44590.0,44050.0,44452.0,541.3003276299921]},
{"i":1,"v":[1648238400.0,44452.74643167,44636.0,44275.0,44337.0,318.5197989300029]},
{"i":2,"v":[1648252800.0,44336.0,44477.0,44112.0,44441.0,193.30515490000215]},
{"i":3,"v":[1648267200.0,44434.07106376,44559.28387061,44379.0,44534.02163765,236.48303291000155]},
{"i":4,"v":[1648281600.0,44521.0,44598.0,44329.0,44335.0,158.88092032999873]},
{"i":5,"v":[1648296000.0,44335.0,44406.0,44165.0,44241.0,146.44034252999973]}],
"ns":{"d":"","indexes":[]},"t":"s1","lbs":{"bar_close_time":1648310400}}},

What the eff does that stuff mean? Let me elaborate.

The first line : 1648224000.0,44345.51290507,44590.0,44050.0,44452.0,541.3003276299921 — Unix timestamp, Open, High, Low, Close, Volume
So that timestamp would be – Friday March 25th 2022 11am DST. There’s a lot of numbers after the decimal, I only need two — so that entry would really read:
Open: 44345.51, High: 44590.00, Low: 44050.00, Close: 44452.00 Volume: 541.30
I was able to specify 240 minute (4-Hour) bars which makes it pretty easy to get a whole day in just a few entries, since 6 bars = 24 hours. Seems easy, right? All I need to do now is to parse that and write it in a way that makes sense for my encoder.

Here it is looking a bit more formatted:

Index,Date,Open,High,Low,Close,Volume
0,"03/25/2022, 03:00:00",43911.22112839,44654.67307908,43606.0,44600.0,915.298048519977
1,"03/25/2022, 07:00:00",44601.0,45082.0,44236.0038521,44346.0,1818.5557992299166
2,"03/25/2022, 11:00:00",44345.51290507,44590.0,44050.0,44452.0,541.3003276299921
3,"03/25/2022, 15:00:00",44452.74643167,44636.0,44275.0,44337.0,318.5197989300029
4,"03/25/2022, 19:00:00",44336.0,44477.0,44112.0,44441.0,193.30515490000215
5,"03/25/2022, 23:00:00",44434.07106376,44559.28387061,44379.0,44534.02163765,236.48321299000156
6,"03/26/2022, 03:00:00",44521.0,44598.0,44329.0,44335.0,158.88092032999873
7,"03/26/2022, 07:00:00",44335.0,44406.0,44165.0,44211.0,196.8329842700001
8,"03/26/2022, 11:00:00",44207.0,44472.0,44152.89142732,44367.0,138.23807569000022
9,"03/26/2022, 15:00:00",44364.0,44785.0,44257.0,44473.0,372.51797744998487

Next steps will be getting more than one quote at a time, which should be possible. In the end I’ll have quite a few of them interleaved among each other, which means I need to lean hard on regular expressions to sift through the sea of data.

(Some time later)

I’m deep into Regular Expressions, a way to sift through the alphabet soup of data and pick out the things that I want. There’s some nice tools out there to help, like regex101 dot com, but its still pretty arcane syntax-wise.
Making some progress, but its getting tricky. Let me explain. I’m using websockets, so I see the data coming from the server and it gets dumped to the console. Problem is, using Regex means it parses whatever it gets its little grubby hands on, which means it could be influenced by debug messages I dump to the console too – a bit like double-dipping into a stream.
So I have to figure out how to debug the program without messing up the datasource. Or at least I think I do at this point. Its messing with my head 🙂
I might be able to get ahead of it by flagging my debugging messages in a way so that it will ignore that, but work on the other data. Maybe… or… split out the results and save them to a file so it doesn’t “pollute” the same stream of data I’m trying to parse.

(Which is really what I should be doing, I think.)

Hoo boy, my head hurts. But I think I have it finally.

This is the data that I’ve been dealing with — just so you have an idea what it looks like raw from the websocket itself:

quote_session ID generated qs_yngjrgxzshkg
chart_session ID generated cs_vfiuozaqmkwh
~m~361~m~{"session_id":"<0.18544.193>_sfo-charts-18-webchart-5@sfo-compute-18_x","timestamp":1648480416,"timestampMs":1648480416379,"release":"registry.xtools.tv/tvbs_release/webchart:release_205-53","studies_metadata_hash":"79c6b847bdfc53283f5b5f6e28f71f7baa91e9f2","protocol":"json","javastudies":"javastudies-3.61_2183","auth_scheme_vsn":2}

~m~484~m~{"m":"qsd","p":["qs_yngjrgxzshkg",{"n":"BITFINEX:BTCUSD","s":"ok","v":{"volume":4537.63954264,"update_mode":"streaming","type":"crypto","short_name":"BTCUSD","rtc":null,"rchp":null,"pro_name":"BITFINEX:BTCUSD","pricescale":10,"original_name":"BITFINEX:BTCUSD","minmove2":0,"minmov":1,"lp_time":1648480412,"lp":47748.0,"is_tradable":true,"fractional":false,"exchange":"BITFINEX","description":"Bitcoin / Dollar","current_session":"market","currency_code":"USD","chp":2.0,"ch":935.0}}]}~m~65~m~{"m":"quote_completed","p":["qs_yngjrgxzshkg","BITFINEX:BTCUSD"]}

So after many attempts that failed, I finally came up with some regex that could filter it into this:

BITFINEX:BTCUSD
Volume: 4537.63979476
Price: 47748.46935417
BITFINEX:BTCUSD
Volume: 4537.67328922
Price: 47748.0
BITFINEX:BTCUSD
Volume: 4537.94068011
Price: 47734.0
BITFINEX:BTCUSD
Volume: 4538.05830511
Price: 47737.0
BITFINEX:BTCUSD
Volume: 4538.90654622
Price: 47732.0
BITFINEX:BTCUSD
Volume: 4539.06445621
Price: 47727.62184819

It took quite a bit to get that all working. Here’s a sample of some of the regex I used:

priceRegex = '\"lp\":(\d+.\d+)'

Make sense to you? Me either, which is why I’m super-glad that sites like regex101 dot com exist. Next, I’ll have to figure out duration between two dates in order to calculate how much quote data to ask for historically.

Until next time…

VTF – Signs And Quote Data

Modding

I said I’d fix all the quoteboard signs, and I did – took a bit of work since I had to do all the contract names and the months (for futures), but it was worth it. Here’s the final result:

Note the visual “weight” is all consistent now. I love the results! Not displaying months yet, but they’re done and look good too.

I also figured out how to implement a “style” layer for colors on the different columns, which allows me to do different things per display element.

Multi-color! Looking pretty good – and necessary for some elements I’ll be displaying, like volume.

A quoteboard is useless without data. I’ve been displaying the same test string over and over just to get the sprite drawing done right, but now its time to get the real deal. I’m doing something a bit unorthodox, since Teardown doesn’t allow you to do direct file read/writes. (For security purposes, which I understand.)

Funny thing, while I was working on this I realized that some of the quote data I wanted to display would take one more column to do so. I’m glad I caught that early, because it would’ve been painful to rework all the boards later on when they had surrounding structures and things. Its always the details that bite you if you’re not careful.

This will probably take more than one post, but I wanted to outline my meandering path towards figuring out how to get some data to display on the boards. As I mentioned, my method for importing data into Teardown is unorthodox, since I’m doing an “out-of-band” method to encode data into vox models.

So, where to get data?

My first thought was using some publicly available services that have some limited free data, using an API (Application Programming Interface) key. I futzed around with a few, but that approach rubbed me the wrong way because it seemed really easy to run up against their query frequency limits.

I wasn’t trying to do anything TOO crazy, but even a moderate polling interval would make it so I’d run up on their limit, and encroach into territory that required paid services. I’m sure that design decision was intentional on their part – not that I blame them, really.

While doing quote source research I realized the big “SPOOOOS” contract had been delisted at the CME. They started trading on April 21st, 1982 and were delisted in September 17th, 2021 – a total of 39 years! I was present on the floor for some of those years, so that hit me pretty hard. I guess the E-Mini was more popular, since its still active. Rest In Peace, spoooos! (We called them that on the floor, probably because when it was september the contract month code is “U”, so SPU sounds like Spoooos.)

Finding ticker data sources is easy, the problem is whether you want to pay $1 – 2,000 USD (per year, about $100/mo) for a full range of data or scrape it from somewhere that has it already. Since this is a hobby project, I’m going to scrape some free sources instead. I need a combination of historical data – so I can get 24/hr and all-time highs and lows as well as current open/high/low/close stuff, and a method to get direct live quotes (semi-delayed is fine) for when I’m updating the boards in real-time mode.

One source I considered was Tradingview.

You know when you have what you think is a clear goal and you just need to achieve one more step? Well, I went down a total rabbit hole when it came to Tradingview and its streaming quotes. I found some Python code “in the wild” that allowed negotiating with their websocket to grab quotes – it was not-so-helpfully formatted like this:

~m~147~m~{"m":"qsd","p":["qs_ofmdqrghftjd",{"n":
"CME_MINI:ESM2022","s":"ok","v":{"volume":442387,
"lp_time":1648220119,"lp":4533.25,"chp":0.46,
"ch":20.75}}]}

All I cared about is getting the “lp” which was “last price” and the volume. Though the timestamp was helpful and the “chp” (Change percentage) and “ch” (Net change) was a nice added bonus. However, I needed more than just one instrument at a time, which required some more Python-ing.

More to come…