Speeding scripts up

[Mikero]

Am posting this here in lieu of anywhere else. The following is a tiny information tute on scripts in general and how to speed them up. All comments welcome, and probably the most important comment would be if this is useful enough to warrant an upload as a submisssion. I dont believe this knowledge is generally known

moderators forgive me if I've got the wrong thread.
----

The ofp engine reads a script (.sqs) or function (.sqf) file once, for it's lifetime. It retains a copy of that script 'in memory' until it encounters an exit statement or equivalent. Only after this point, if the script is required again, is it re-read from file. Exit statements in themselves therefore, induce lag wherever a file read is caused.

At no time does the engine retain a compiled version of the script. Each command is individually processed as and when. This is termed interpreting in the trade (as opposed to compiling).

For the lifetime of a script, any 'goto' style command causes the script to be re-interpreted from it's beginning to wherever the #target is. Read that statement carefully and dismiss any ideas that it sometimes 'jumps ahead' or 'skips' or does something special. Any goto, causes an unconditional re-interpret, from the beginning. Dismiss your ideas of how this works because they caused you lag.

For this reason, large initialise scripts can measurably cause lag. How much lag is the same as a length of string, but it would be perceptible. The reason is of course that useless text is re-'interpreted' and discarded.

The traditional approach in almost all scripts is as follows
//////////////////////
do lots of initialise
...........
and lots more

#loop
.....
goto loop
//////////////////////
to speed scripts up

goto initialise

#loop
.....
goto loop

#initialise
........
goto loop

putting any necessary exit commands as appropriate of course, but the intent here is to get the #targets as close to the top as possible.

[UNN]

I understand your point, and your suggestion wont harm even if it's not the case, so I will probably follow that format. But how can you be so certain that it does re-interpret at every goto. Is it possible to test?

I would not have thought twice about it, until I started looking through the save game files using CoC's BinView. You can view each script, what point it was at when saved and what values where in the variables.

If OFP organises the scripts in memory the same way it does in the saved game, then there is nothing to worry about?

[macguba]

I'll leave this thread open because I'm sure it will stimulate some intelligent debate.  

@mikero please write this up as a wee tutorial and submit it to the Ed Depot.    

Information at OFPEC is always stored in the Ed Depot, not the forums.

[Mikero]

@UNN

>wont harm even if it's not the case

wisest remark anyone can make.

>How certain are you?

100% cast iron, gold plated, set in concrete garantee that a script file is re-interpreted from the 'top of file'. I am trying to use great caution in not inferring the file is re-read (another common misconception)

One test, albeit not conclusive is

#loop
----
goto loop
----
#loop
----
goto loop

very first #loop, is the #loop. There are other tests and it has been thrashed around over time, in the forums. I am unable to find the relevant threads right now, and prefer not to as there is additonal 'noise' in them which only serves to confuse the issue (or at least blurr it)

I am not familiar with the coc scripts and never have had success with that excellent tool, binview when reading fps files. What I can tell you is that the stored contents of these, and so-called 'binary' mission.sqm's, config.cpp's and description.ext's are in fact tokenised versions of the scripts where, simply stated, any repetetive text strings are only held once and tokens used to access the same string. They are not binary or compiled in the normal sense of the word.

I have a little blurb on that subject.

But, you definitely get the last word of wisdom here, it can't possibly be worse, and is likely to improve things.

[Terox]

I didnt know this, so as per the above post, i ran the following script

_count = 0
#LOOP
_count = _count + 1
~1
hint "first loop"
if(_count >1)then {goto "LOOP"}
#LOOP
~2
hint "second loop"
goto "LOOP"

and the first "Loop" label was used, not the second

Now my evaluation on this is either
a) The label is searched from the start of the script (most likely)
b) the position of the label is kept in memory somehow maybe via line number(doubtful)


anyway, from now on all my scripts will be wrote with goto "INIT" as the first line and then the actual main loop will be at the top of the script


 
Thx for the info

[UNN]

From the point of view of, best practice. Pushing the init routines to the end, is not up for debate. But I was curious about testing it, and just understanding more about how OFP handles scripts. So if this is offtopic, my apologise.

In BinView, labels are stored as individual items, and appear to be given an index by OFP.



Again in BinView, each line of code is stored just above the labels. In this case the repeat loop is right at the end:



The line the Repeat loop points to (32_BIT_SIGNED 16) is here:



If I can track down and change the Label index I should be able to test it. In theory I should be able to save a script before it reaches the last label, then hex edit the saved game to point to another label. If the script executes as normal and ignores my changes, then that proves the saved game format has no relation to the script in memory. If it does respond to my changes then we know OFP does not re-read every line, it still probably has to loop though the script to find index 16. So either way, you might be reducing potential lag by putting the init stuff at the end, and loops at the start.

Ok so this is definetly off topic, WaitUntil is used by @ and suspend until is used by ~.

But quickly back on topic again To make life that bit easier, you can setup Chris' OFP Script Editor with a suitable template:

Code: [Select]

;Initialise the script
goto "Init"

;Start the main body
#start



;End the script here
Exit

;Initialise the script here
#Init

_This Select 0

;Return to start
goto "Start"

[Mikero]

wow UNN you made that easy to understand!

My best, almost certain, bet, is that if you alter via a hexedit, the changes will occur in the 'game'.

This because, only non-exiting scripts, as in quote "one's in use at the time of the save" are actually saved in the fps.

that might incidentally, startle you, perhaps not. To give an example

fred.sqs

_thing=50;
exit

thing.sqs will not be found in the fps file (unless by extraordinary coincidence it happened to be 'in use' at the time.

Editing script files 'on the fly' can only happen if they are exiting scripts. One's that stay resident, eg anything on a permanent loop cannot. And the only way to change their behaviour is to hexedit. The fps file for this purpose, is much the same as saying 'resident in memory'.

This is not off topic for me because it leads back to the statement that exiting scripts, by their nature, are lag inducing (they cause a re read, re-binarise, of the 'file')

As for the very interesting decodes in your binview the 'strings' have associated with them an 'index number' a value you can clearly see in your example. It is these index numbers and these strings which make up the so-called 'binary' mission.sqms and are the object of affection in Amalfi's bin2cpp and cpp2bin utilities.

For me, the phrase re-interpret, means re-interpreting the strings present in the above tables you illustrate. Admittedly, considerably less cpu crunch overhead in interpreting them opposed to stripping out semicloons tabs and newlines from a text file, but re-interpreted all the same. There is no hash index construct to go directly to a given line entry. A shame.

So, now I guess it is wandering off topic and I'll take Macguba's suggestion to upload it as a submission.

Thank you all, for your input. I can see it's already been beneficial for people.

[Terox]

Ref the "Exit"
could you explain this more

are you saying that

The following examples:
 A ( a looping script)

goto "INIT"

#Loop
~1
XXXXXX
XXXXXX
XXXXXX
XXXXXX
If(XXX)then{goto "LOOP"}else{exit}

#INIT
XXXX
XXXX
goto "LOOP"

 and
 B (a linear script[/b][/color]

XXXXXX
XXXXXX
XXXXXX
XXXXXX
exit

are better of being re written in the following way

goto "INIT"

#Loop
~1
XXXXXX
XXXXXX
XXXXXX
XXXXXX
If(XXX)then{goto "LOOP"}

#INIT
XXXX
XXXX
if(_time <1)then{goto "LOOP"}

 and
 B (a linear script[/b][/color]

XXXXXX
XXXXXX
XXXXXX
XXXXXX
;;;;; exit

surely it must use some process to see that the EOF has been reached, i could see the advantage of not using an exit on a linear script, but a script with a loop, the only place you can really "goto"for an EOF, is a label at the end of the script , which means the script must be "re read, a line at a time to get to the label and therefore the difference in cpu requirements must be minimal or as my example below, allow the script to rerun the ~init section and not loop using a _time query (which may be impracticle in a large script with lots of loops), which again, then by your above findings suggests that all larger scripts should be reduced into several small scripts, as many as possible being run in a linear system calling upon the next script in the chain as the previous one completes


If you could expand on the none use of the "exit" command, i would be most appreciative

infact just an overall good and bad practices, on an advanced level, would be very welcome

[Mikero]

ooooooo sorry for that, just shows how careful I have to be with phrasing things.

the exit statement or equivalent is the definition of a non resident script. One that will, by it's nature will cause a file access to REoccur if it's called again, because it is dumped from memory as no longer required. Resident scripts do a single file read in contrast to this.

I did not mean to imply (as I did do) that the exit function itself was a criminal.

Nor do I have remedies to prevent file reads. The point here was to enable folks to understand that goto statements cause scripts to be reINTERPRETED, not re read from file (which has often been the assumption)

Your script examples are 'correct'. The emphasis here is to always put #targets as close to top of file as possible.

[Terox]

so , if i've got this right

"exit" has the same effect on a script

that
myvariable = nil
has

so as basic coding practice
use EXIT for only run once type scripts (eg INIT.sqs or an INTRO)
and
Omit EXIT if the script will be called to run more than once (eg from an eventhandler)

which probably means that multiple calls on a looping script would be better having a goto "End" and have that label at the very bottom of the script, rather than an exit somewhere

One thing i am curious about though

if a script has no more lines of code to be read, how does the "read script" system know when to stop looking for lines of code

[Baddo]

One thing i am curious about though

if a script has no more lines of code to be read, how does the "read script" system know when to stop looking for lines of code

As far as I know anything, OFP will do it just like any other program written in C++.

Something like:

Code: [Select]

...
string s;
while (getline(fin, s))
{
    cout << s << endl;
}
...
will see when the EOF arrives and stops there. Why? Because C++ functions cin and getline set the eofbit (and in some cases, also failbit) to 1 when they arrive at the end of the file. This is just an example, I am not saying cin or getline is used for the job but the same applies to other methods too.

From some website:

Operating systems need to keep track of where every file ends. There are two techniques for doing this: One is to put a special end-of-file mark at the end of each file. The other is to keep track of how many characters are in the file.

So, when OFP reads a file stream, it encounters a special mark in the file and knows to stop there.

[Mikero]

@Terox

sorry, you got it wrong. You're focussing on the wrong end of the barrel (exit)

which probably means that multiple calls on a looping script would be better having a goto "End" and have that label at the very bottom of the script, rather than an exit somewhere

'multiple calls' automatically implies that the script being called has, one way or the other, exited. Stopped, ceased to exist, no longer running, not resident in memory. You wouldn't be calling a still active looping script.

'goto end' would be the worst possible outcome because it would cause the ENTIRE script to be re-interpreted to it's END just to exit (gasp). In fact, this is a perfect example of when you *would* use an exit statement there and then, to cease and desist.

also 'event handlers' calling looping scripts is a shocking crime punishable by measles, the pox and hopefully bouts of dysentry. It is the worst lag demon of them all (unless large ~waits are used). An 'event' is an interrupt. Something you want to have handled quickly and then get the hell out of there, quickly.

@baddo said it all, an end of file (EOF) is an implied exit.

[THobson]

I have never written a code interpreter, but if I were to do such a thing I think I would have code that:
1. stripped out all the leading blank characters from a line
2. read the next character.
3. Take any appropriate action

In OFP the first non-blank characters of lines can have special meanings ( ; @ # ~ &)  So to find a lable it would be necessary initially to find the first non-blank character of a line == # then check to see if this is the target lable.

If that is how it is done then things that would effect the performance of a goto lable combination would be:

The number of lines above the target lable
The number of blank characters at the start of all the lines above the target lable
The number of lables above the target lable.

Is this making sense?

[macguba]

Gentlemen,

When this discussion has run its course, it would be good if somebody could write up the conclusions into an "idiot's guide to making your scripts run faster" tutorial.     You don't need to understand the technicalities to benefit from simple advice like using "exit" directly rather than "goto #end".

[KTottE]

A lot of the scripts I see, and some of the ones I've written myself, use labels for flow control.

? (expression) : goto "label"
? (other_expression): goto "other_label"

How bad is this, assuming that the checks are done at the top of the file? Would performance be gained by using if {} then {}-statements instead?

What is the relative performance hit of instantiating new scripts as opposed to jmp:ing to labels? Would the first example be better than the second:

Code: [Select]

; first.sqs
? (expression): [] exec "second.sqs"

; second.sqs
; Do Magic.


Code: [Select]

? (expression): goto "second"

; Lots of stuff here

#second
; Do Magic.

Thanks for bringing this to our attention, Mikero, I know of a lot of scripts that need rewriting