-
September 25th, 2020, 15:22 #1
Floating Point Precision differences
I'm just about done a big update to one of my extensions and went back to test it in Classic and one of the features is totally nonfunctional (but works beautifully in Unity).
Has the floating point precision been upgraded in Unity?
I did try to do a search for this, but mostly found old threads on the 'issue' in FGC.bmos' extensions
he/them
-
September 25th, 2020, 16:14 #2
Supreme Deity
- Join Date
- Mar 2007
- Posts
- 20,540
Not that I'm aware of. Both FGC and FGU use "double" data types for numbers. However, they do use different implementations of almost every internal function (C# vs. C++). I'd need more information on what APIs are reporting differently.
Regards,
JPG
-
September 25th, 2020, 18:06 #3
Here is a simple function to demonstrate:
Code:local function onTest() local nBigNumber = 12345678910 Debug.chat('nBigNumber: ' .. nBigNumber) Debug.chat(nBigNumber) end
classic.jpg
In Unity:
unity.jpg
This might actually just be something different about how debug.chat works, but it's really been making my head hurt!
Based on this testing, it looks like the actual value is still there (since it shows when concatenated into a string).
I will use the approach of storing the number in a string for now so I can see what is actually happening in my extension's code.Last edited by bmos; September 25th, 2020 at 18:26.
bmos' extensions
he/them
-
September 25th, 2020, 18:29 #4
Zygmunt Molotch might have just given me the answer.
Apparently lua has 64-bit libraries that allow extra precision.
Perhaps you have this in Unity and not in Classic?bmos' extensions
he/them
-
September 25th, 2020, 18:47 #5
Supreme Deity
- Join Date
- Mar 2007
- Posts
- 20,540
Not sure. I believe they use the same libraries; but the interface to the libraries is 32-bit on FGC and 64-bit on FGU. That may or may not be the issue. "double" is a 64-bit data type in both C++ and C#; and shouldn't vary between 32-bit vs. 64-bit libraries.
The Debug output may just be different; since the code is very different between FGC/FGU.
The real question is where the number differences are showing up in the code that is causing issues. (number -> string conversion, API return value precision, etc.)
Regards,
JPG
-
September 25th, 2020, 20:07 #6
Last edited by bmos; September 25th, 2020 at 20:13.
bmos' extensions
he/them
-
September 25th, 2020, 20:27 #7
Alright, I found where the two are behaving differently.
Code:DB.setValue(window.getDatabaseNode(), 'starttime', 'number', nPreciseTime) Debug.chat('set: ' .. nPreciseTime, 'check set: ' .. DB.getValue(window.getDatabaseNode(), 'starttime', 0))
I have no handlers associated with that node. The code I included is in the script tag on a buttoncontrol object (no extra templates). I tried changing 'starttime' to another name that I have not used before to verify that nothing in my code was targeting that node and it behaved the same way.
In Unity, these numbers match.Last edited by bmos; September 25th, 2020 at 20:34.
bmos' extensions
he/them
-
September 25th, 2020, 21:47 #8
Supreme Deity
- Join Date
- Mar 2007
- Posts
- 20,540
Ah, found it. FGC stores numbers has single floats internally; and FGU was upgraded to use double floats internally.
Regards,
JPG
-
September 25th, 2020, 21:48 #9
Supreme Deity
- Join Date
- Mar 2007
- Posts
- 20,540
You might need to divide before you can save the number to write code you can use in both.
Regards,
JPG
-
September 25th, 2020, 22:41 #10
Ok! Thank you so much.
EDIT: For anyone who ends up here with the same issue in the future, here is an even better workaround (in some ways):
Save it as a string in the database with DB.setValue(parentnode, 'nodename', 'string', tostring(nVar))
Then, on the other end, use tonumber(DB.getValue(parentnode, 'nodename'))
There is still a potential for some inaccuracy (according to a friend of mine, at least), but not nearly as much as single-precision floating point.Last edited by bmos; September 26th, 2020 at 18:27.
bmos' extensions
he/them
Thread Information
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)
Bookmarks