On Wed, 19 Jun 2024 15:02:28 -0000 (UTC), Rich wrote:

> Luis Mendes <luisXXXlupeXXX@gmail.com> wrote:
>> Hi all!
>>
>>
>> My program is working fine when thread::send don't use the -async
>> option. When it does, all of those created threads exit prematurely.
>>
>> The pseudo-code I have is this:
>
> Working code that you've tested to exhibit the bug you see is
> preferable, and your code was *very* close....
>
>> ===== main file
>>
>> while 1 {
>> ...
>> while {nr_live_threads < nr_max_threads} {
>
> This will error out as a syntax error. You want to both initialize
> these variables before you use them, and to interpolate them using $
> above.
>
>> set tid [thread::create $init_script] thread::send -async $tid
>> [list sourceFiles ....]
>> }
>> after 10000
>> }
>
> You never increment nr_live_threads, so this loop above will (assuming
> the variables were initalized, and referenced, correctly) simply loop
> forever, creating new threads. At least until the whole process is
> killed for using all free memory up.
>
>> ===== oo.tcl
>> namespace eval ns0 {
>> proc runAnsible {...} {
>> Parse new ... vwait ::exit_flag
>> }
>> }
>
> You never signal to the master thread that this thread has exited, so
> the master (as written here) will never launch a new thread when an
> existing one finishes.
>
>> This comprises the important parts of the script, I think.
>> When thread::send does not use `-async`, the `vwait ::exit_flag` works
>> and the thread is run until the end.
>> With `-async`, the thread exits shortly after the `thread::send`
>> command.
>
> Something must be different in the "psudeo" code vs. your real code
> then.
>
>> I've read about `thread::preserve` and `thread::release`, but
>> interpreted it as necessary when threads have to be orchestrated and
>> some may be dependent on the results of others.
>
> No, those are to do reference counting for thread cleanup.
>
>> What I want is really to have several threads launched in the same
>> moment,
>> at each run of the while loop that checks if the number of active
>> threads is less than the nr_max_threads.
>> How can that be accomplished?
>
> Well, first, you have to communicate the exit of a child thread back to
> he main thread, and have that comm path decrement "nr_active" (and you
> also need to increment nr_active when you launch a new thread).
>
> Syntax cleaned up -- and simplified version of your original code, that
> *actually runs*:
>
> thread-test:
> #!/usr/bin/tclsh
>
> package require Thread
>
> set nr_max_threads 4 set nr_live_threads 0
>
> set init_script {
> puts stderr "Thread: [thread::id] Init: creating sourceFiles"
> proc sourceFiles {args} {
> source oo.tl ns0::runAnsible $args
> }
> puts stderr "Thread: [thread::id] waiting"
> thread::wait puts stderr "Thread: [thread::id] out of wait"
> }
>
> while 1 {
> while {$nr_live_threads < $nr_max_threads} {
> puts stderr "Main: live=$nr_live_threads
> max=$nr_max_threads"
> set tid [thread::create $init_script]
> incr nr_live_threads thread::send -async $tid [list
> sourceFiles ....]
> }
> puts stderr "Main: sleeping for 10s"
> after 10000 puts stderr "Main: awake"
> }
>
> oo.tl:
> namespace eval ns0 {
> proc runAnsible {...} {
> puts stderr "Thread [thread::id]: executing parse new"
> Parse new ...
> puts stderr "Thread [thread::id]: vwait begin"
> vwait ::exit_flag puts stderr "Thread [thread::id]: vwait
> complete"
> }
> }
> oo::class create Parse {
> constructor {...} {
> set random [expr {int(rand()*20000)}]
> puts stderr "Thread [thread::id]: object constructor -
> sleeping for $random"
> after $random [list set ::exit_flag 1]
> }
> }
>
> Sample run of the above:
>
> $ ./thread-test Main: live=0 max=4 Thread: tid0x7fbfa7b6c640 Init:
> creating sourceFiles Thread: tid0x7fbfa7b6c640 waiting Main: live=1
> max=4 Thread tid0x7fbfa7b6c640: executing parse new Thread
> tid0x7fbfa7b6c640: object constructor - sleeping for 8476 Thread
> tid0x7fbfa7b6c640: vwait begin Thread: tid0x7fbfa6b6a640 Init:
> creating sourceFiles Thread: tid0x7fbfa6b6a640 waiting Main: live=2
> max=4 Thread tid0x7fbfa6b6a640: executing parse new Thread
> tid0x7fbfa6b6a640: object constructor - sleeping for 16806 Thread
> tid0x7fbfa6b6a640: vwait begin Thread: tid0x7fbfa6369640 Init:
> creating sourceFiles Thread: tid0x7fbfa6369640 waiting Main: live=3
> max=4 Thread tid0x7fbfa6369640: executing parse new Thread
> tid0x7fbfa6369640: object constructor - sleeping for 11225 Thread
> tid0x7fbfa6369640: vwait begin Thread: tid0x7fbfa5b68640 Init:
> creating sourceFiles Thread: tid0x7fbfa5b68640 waiting Main:
> sleeping for 10s Thread tid0x7fbfa5b68640: executing parse new
> Thread tid0x7fbfa5b68640: object constructor - sleeping for 5573
> Thread tid0x7fbfa5b68640: vwait begin Thread tid0x7fbfa5b68640:
> vwait complete Thread tid0x7fbfa7b6c640: vwait complete Main: awake
> Main: sleeping for 10s Thread tid0x7fbfa6369640: vwait complete
> Thread tid0x7fbfa6b6a640: vwait complete Main: awake Main: sleeping
> for 10s Main: awake Main: sleeping for 10s
>
> And, it will continue to loop saying 'awake' and 'sleeping' since the
> exit of the children is never communicated to the master.
>
> You need to master to become aware that one of the children has exited,
> so it knows to relaunch another child. One way is to use the additional
> result variable for -async threads and vwait on that variable in the
> master.
>
> Here is the 'diff' necessary to have the master monitor children exiting
> and to launch a new child when that happens:
>
> --- thread-test.v1 2024-06-19 10:42:34.359605931 -0400 +++
> thread-test 2024-06-19 11:00:01.433949725 -0400 @@ -4,6 +4,7
@@
>
> set nr_max_threads 4 set nr_live_threads 0
> +set sync 0
>
> set init_script {
> puts stderr "Thread: [thread::id] Init: creating sourceFiles"
> @@ -21,10 +22,10 @@
> puts stderr "Main: live=$nr_live_threads
> max=$nr_max_threads"
> set tid [thread::create $init_script]
> incr nr_live_threads
> - thread::send -async $tid [list sourceFiles ....]
> + thread::send -async $tid [list sourceFiles ....] sync
> }
> - puts stderr "Main: sleeping for 10s"
> - after 10000 - puts stderr "Main: awake"
> + puts stderr "Main: waiting for a child to exit"
> + vwait sync + puts stderr "Main: a child exited"
> + incr nr_live_threads -1
> }

Hi Rich,

Once again, thank you very much for your help.
I could manage to application run in a multi-threaded way.

Still, there are a couple of things that I haven't yet understood, maybe
you (or other person) can help me figure this out.

1. Regarding vwait
As stated in https://www.tcl-lang.org/man/tcl/TclCmd/vwait.htm
"""It continues processing events until some event handler sets the value
of the global variable varName. Once varName has been set, the vwait
command will return as soon as the event handler that modified varName
completes."""
This was a difficulty I had before, maybe because English is not my main
language.
I thought that varName would have to change for every event handler that
signaled the end of some operation, like:
- open fd 1
- do work
- set varName 1
then,
- open fd 2
- so work
- set varName 2 (or something different)
just because it's said "the vwait command will return as soon as the event
handler that *modified* varName completes".
That *modified* make me think of that, but it seems that for both
operation it's enough to set varName 1 in both, right?

2. Regarding the script that you modified, I changed it a bit as well to
show what I don't understand.
The thread::names command shows all the threads that have been created
from the start of execution and not only the ones that were created in the
last cycle.
And this is confirmed by thread::exists that show 1 for all of them.
I was expecting that threads would no longer exist after an event handler
sets a varName for vwait.
Otherwise, we can end up with millions of threads existing at the same
time.

At cycle nr. 9:
++++++++ cycle 9
Main: live=3 max=4
Thread: tid0x7f5171ffb6c0 Init: creating sourceFiles
Thread: tid0x7f5171ffb6c0 waiting
Main: waiting for a child to exit.
Main: a child exited.
tid0x7f5171ffb6c0 1
tid0x7f51727fc6c0 1
tid0x7f5172ffd6c0 1
tid0x7f51737fe6c0 1
tid0x7f5173fff6c0 1
tid0x7f5190ff86c0 1
tid0x7f51917f96c0 1
tid0x7f5191ffa6c0 1
tid0x7f51927fb6c0 1
tid0x7f5192ffc6c0 1
tid0x7f51937fd6c0 1
tid0x7f51947ff6c0 1
tid0x7f5194b10400 1

Here's the script for thread_test.tcl.
I didn't change oo.tcl.

"""
#!/usr/bin/tclsh

package require Thread

set nr_max_threads 4
set nr_live_threads 0
set sync 0

proc anykey {{msg "Hit any key: "}} {
set stty_settings [exec stty -g]
exec stty raw -echo
puts -nonewline $msg
flush stdout
read stdin 1
exec stty $stty_settings
puts ""
}

set init_script {
puts stderr "Thread: [thread::id] Init: creating sourceFiles"
proc sourceFiles {args} {
source oo.tcl
ns0::runAnsible $args
}
puts stderr "Thread: [thread::id] waiting"
thread::wait
puts stderr "Thread: [thread::id] out of wait"
}

set cycle 0
puts "Starts with::[thread::id]"
while 1 {
puts "++++++++ cycle [incr cycle]"
while {$nr_live_threads < $nr_max_threads} {
puts stderr "Main: live=$nr_live_threads max=$nr_max_threads"
set tid [thread::create $init_script]
incr nr_live_threads
thread::send -async $tid [list sourceFiles ....] sync
}
puts stderr "Main: waiting for a child to exit."
vwait sync
puts stderr "Main: a child exited."
foreach tn [thread::names] {
puts "$tn\t\t[thread::exists $tn]"
}
incr nr_live_threads -1
anykey
} """

Best regards,

Luis