Any ideas on the cause of this error below?
It happens during a http request to a local machine in the LAN.
I cannot debug this, since it happens on a machine of a customer.
It works for me, when I connect to the same machine over VPN.

too many nested evaluations (infinite loop?)
while executing
"expr {int(100.0 * double($received)/double($expected))}"
(procedure "::UIProgressBar" line 5)
invoked from within
"::UIProgressBar ::http::9 5413377 2244608"
("eval" body line 1)
invoked from within
"eval $state(-queryprogress) [list $token $state(querylength)
$state(queryoffset)]"
(procedure "http::Write" line 88)
invoked from within
"http::Write ::http::9"

The http request is hooked up to a callback procedures that updates a
progress bar variable:

## Update the variable of the main progress bar when uploading or
downloading files.
# \param token token of the http transaction if this progress bar
reflects e.g. an upload or download process
# \param expected the amount of bytes expected in case of an upload or
download process. Any integer number in other cases.
# \param received the amount of bytes received in case of an upload or
download process. Any integer number in other cases.
# \return
proc ::UIProgressBar {token expected received} {
if {$expected==0} {
return
}
set ::progress [expr {int(100.0 *
double($received)/double($expected))}]
update
}

Many thanks for any hints or advices.
Regards
Alexandru

alexandru <alexandru.dadalau@meshparts.de> wrote:
> Any ideas on the cause of this error below?
> It happens during a http request to a local machine in the LAN.
> I cannot debug this, since it happens on a machine of a customer.
> It works for me, when I connect to the same machine over VPN.
>
>
> too many nested evaluations (infinite loop?)

This is Tcl's "infinite recursion detected" error message.

> The http request is hooked up to a callback procedures that updates a
> progress bar variable:
>
> ## Update the variable of the main progress bar when uploading or
> downloading files.
> # \param token token of the http transaction if this progress bar
> reflects e.g. an upload or download process
> # \param expected the amount of bytes expected in case of an upload or
> download process. Any integer number in other cases.
> # \param received the amount of bytes received in case of an upload or
> download process. Any integer number in other cases.
> # \return
> proc ::UIProgressBar {token expected received} {
> if {$expected==0} {
> return
> }
> set ::progress [expr {int(100.0 *
> double($received)/double($expected))}]
> update

This update is reentering the http module (it uses event driven code
internally), which then calls ::UIProgressBar again, before the first
update completes. Which then repeats the process again, until Tcl
detects the excessive recursion and issues the error message you quoted
at the start.

Thanks for the explanation.
Though I still don't understand why is this happening.
Why is the callback started before the first one finishes?
How can I avoid this happening?
Why is this happening only in one case?

* alexandru.dadalau@meshparts.de (alexandru)
| Thanks for the explanation.
| Though I still don't understand why is this happening.
| Why is the callback started before the first one finishes?

start:
Because you call [update] in the callback. This enters the event loop,
which ends up in your callback
- goto start:.

| How can I avoid this happening?

By not calling [update].

The [update] should happen anyway when the application becomes idle.

If the input comes too quickly, so the handler is called with no pause,
you could experiment with
update idle
or
after 0 update
or something like this. Though it might be a good idea to keep track
whether an update has already been scheduled etc...

| Why is this happening only in one case?

Might be a timing effect, and exactly which amount of input is available
at the call of [update].

R'

Thanks Ralph, I get it now.
I'll experiment with your suggestions.

Regards
Alexandru

On 5/29/2024 1:34 PM, alexandru wrote:
> Thanks Ralph, I get it now.
> I'll experiment with your suggestions.
>
> Regards
> Alexandru

I'm having too much fun with chatGPT. I first asked it about calling *update* in a progress bar callback and it gave me an example of what can go wrong.

So, I asked it to create a *ttk::progressbar* example where each click of a button would invoke a callback that incremented the progress variable by 10% but would *not* call update.

Getting greedy, I asked it to provide a percent done indicator (I didn't say use a label) and then also bind the mouse-wheel so that each wheel-click over the button would increment or decrement it by 5%.

After re-writing the code for each of my added requests, it finally said: You’ve reached your GPT-4o limit.
Responses will use GPT-3.5 until your limit resets after 9:51 PM (in 4 hours). I didn't want to pay for the upgrade (:

Anyway here's the code it wrote, nicely commented, and I pasted it into a windows console and it ran perfectly.

I'm thinking there has to be a Tcl programmer on a treadmill somewhere creating this code :)

package require Tk

# Procedure to create and initialize the progress bar
proc create_progress_bar {title max_value} {
global progress_var max_progress_var increment_value percent_label

# Create the main window
set win [toplevel .progress]
wm title $win $title

# Create a label
label $win.label -text "Progress:"
pack $win.label -padx 10 -pady 5

# Create a variable to hold the progress value
set progress_var 0
set max_progress_var $max_value

# Calculate the increment value as 10% of the maximum value
set increment_value [expr {round($max_value * 0.10)}]

# Create a progress bar widget
ttk::progressbar $win.progress -variable progress_var -maximum $max_value -length 300
pack $win.progress -padx 10 -pady 10

# Create a label to show the percentage done
set percent_label [label $win.percent_label -text "0%"]
pack $percent_label -padx 10 -pady 5

# Create a button to increment the progress bar
button $win.button -text "Update Progress" -command update_progress
pack $win.button -padx 10 -pady 5

# Bind mouse wheel events to the button
bind $win.button <MouseWheel> "mouse_wheel_scroll %D"
}

# Procedure to update the progress bar by button click (10%)
proc update_progress {} {
global progress_var max_progress_var increment_value percent_label

# Increment the progress variable by 10% of the maximum value
set progress_var [expr {$progress_var + $increment_value}]

# Ensure progress_var does not exceed max_progress_var
if {$progress_var >= $max_progress_var} {
set progress_var $max_progress_var
.progress.button configure -text "Done"
}

# Update the GUI to reflect the progress change
update_progress_bar
}

# Procedure to handle mouse wheel scroll
proc mouse_wheel_scroll {delta} {
global progress_var max_progress_var

# Calculate the increment value as 5% of the maximum value
set increment_value [expr {round($max_progress_var * 0.05)}]

# Increment or decrement the progress variable based on scroll direction
if {$delta > 0} {
set progress_var [expr {$progress_var + $increment_value}]
} else {
set progress_var [expr {$progress_var - $increment_value}]
}

# Ensure progress_var is within valid range
if {$progress_var > $max_progress_var} {
set progress_var $max_progress_var
} elseif {$progress_var < 0} {
set progress_var 0
}

# Update the button text if progress reaches 100%
if {$progress_var == $max_progress_var} {
.progress.button configure -text "Done"
} else {
.progress.button configure -text "Update Progress"
}

# Update the GUI to reflect the progress change
update_progress_bar
}

# Procedure to update the progress bar and label
proc update_progress_bar {} {
global progress_var max_progress_var percent_label

# Calculate the percentage done
set percent_done [expr {int(($progress_var / double($max_progress_var)) * 100)}]

# Update the label with the current percentage done
$percent_label configure -text "$percent_done%"

# Update the progress bar widget
.progress.progress configure -value $progress_var
}

# Main script to run the example
create_progress_bar "Progress Bar Example" 100

# Start the Tk event loop
tkwait window .progress

Thanks, but what is the point in updating a progress bar only when the
user makes a click?

* alexandru.dadalau@meshparts.de (alexandru)
| Thanks, but what is the point in updating a progress bar only when the
| user makes a click?

Good question :-). Interactive events are bound to happen in at least
milliseconds intervalls, so there is aeons of time for the computer to
update the screen, thus [update] is not necessary at all.

The problem usually is that the events happen so quickly that the 'idle'
handler (which does the screen update in Tk) has no chance to run.

Consider a 'readable' fileevent on a regular large file: this will
trigger repeatedly until EOF, and any change to widgets (eg, "read N%
complete") will not be visible until EOF, where it jumps from 0% to 100%.

R'

On 5/30/2024 5:39 AM, Ralf Fassel wrote:
> * alexandru.dadalau@meshparts.de (alexandru)
> | Thanks, but what is the point in updating a progress bar only when the
> | user makes a click?
>
> Good question :-). Interactive events are bound to happen in at least
> milliseconds intervalls, so there is aeons of time for the computer to
> update the screen, thus [update] is not necessary at all.
>
> The problem usually is that the events happen so quickly that the 'idle'
> handler (which does the screen update in Tk) has no chance to run.
>
> Consider a 'readable' fileevent on a regular large file: this will
> trigger repeatedly until EOF, and any change to widgets (eg, "read N%
> complete") will not be visible until EOF, where it jumps from 0% to 100%.
>
> R'

Correct. That was a poor simulation. What is needed is to have some code that simulates http requests (or file events) that run essentially as though they were 100% compute bound.

This of course then will starve any GUI.

I asked chatGPT how to get around this and it said this:

---------------
Yes, if the progress bar is updated too quickly, such as from very fast HTTP events, it might not display intermediate values and only show 0% and 100%. This happens because the updates are too rapid for the GUI to refresh and display the intermediate states.

To ensure the progress bar displays intermediate values even with very fast events, you can use update or update idletasks to force the GUI to process the pending events and update the display.
--------------

It then wrote code that used update again. So, I asked it to use 2 threads instead. It said:

-------------
Yes, using update in a fast loop can indeed cause potential issues, including the risk of recursive infinite loops if not handled carefully. Using threads is a good alternative to ensure the GUI remains responsive while processing fast events.

Here's how you can implement it using two threads: one for handling the fast events and another for updating the GUI.
------------

After about 7 iterations, where it appeared to me the thread would die silently, and during which I asked it to include puts statements (using send::thread back to the main thread since I'm testing on windows and need the puts to go to the console) it finally got the quoting right. I had to point out the problems with needing escaping and that it needed to define thread_proc inside the thread, something I'm quite familiar with. However, it was very patient and incrementally corrected the code.

Bottom line is it did finally get it to work. It took me about 30 minutes to coax it into a result that worked.

While it (and I) didn't solve the problem in a single thread using the standard Tk event system, I believe it does effectively solve the problem of updating a gui when events can trigger extremely fast.

To fully simulate what the OP wants to do, it would need to do true http events inside a second thread.

I have written code in the past which does this, including where I created dozens of threads that each would do an http request concurrently and it worked just fine. This was part of my "tasks" project I wrote for amusement during covid.

Here's the code it finally wrote using 2 threads. All I added was to close the . toplevel and open a console:

---------------

package require Tk
package require Thread

wm wi .
console show

# Procedure to create and initialize the progress bar
proc create_progress_bar {title max_value} {
global progress_var max_progress_var increment_value percent_label main_thread_id

# Save the main thread ID
set main_thread_id [thread::id]

# Create the main window
set win [toplevel .progress]
wm title $win $title

# Create a label
label $win.label -text "Progress:"
pack $win.label -padx 10 -pady 5

# Create a variable to hold the progress value
set progress_var 0
set max_progress_var $max_value

# Calculate the increment value as 10% of the maximum value
set increment_value [expr {round($max_value * 0.10)}]

# Create a progress bar widget
ttk::progressbar $win.progress -variable progress_var -maximum $max_value -length 300
pack $win.progress -padx 10 -pady 10

# Create a label to show the percentage done
set percent_label [label $win.percent_label -text "0%"]
pack $percent_label -padx 10 -pady 5

# Button to start simulating fast HTTP events
button $win.button -text "Start Fast Events" -command simulate_http_events
pack $win.button -padx 10 -pady 5
}

# Procedure to simulate very fast HTTP events
proc simulate_http_events {} {
global max_progress_var increment_value main_thread_id

# Define the thread procedure script
set script "
proc thread_proc {main_thread_id increment_value max_progress_var} {
# Debugging output to indicate the thread is running
thread::send -async \$main_thread_id \[list puts \"Thread started\"\]
# Simulate fast HTTP events
for {set i 0} {\$i < 10} {incr i} {
# Debugging output for each iteration
# thread::send -async \$main_thread_id \[list puts \"Thread iteration: \$i\"\]

# Send signal to main thread to update the progress variable
thread::send -async \$main_thread_id \[list handle_http_event \$increment_value \$max_progress_var\]

# Pause briefly to simulate processing time
after 100
}
# Debugging output to indicate the thread has finished
thread::send -async \$main_thread_id \[list puts \"Thread finished\"\]
}
thread_proc $main_thread_id $increment_value $max_progress_var
"

# Create a new thread to handle fast HTTP events
set event_thread [thread::create $script]
}

# Procedure to handle HTTP event updates in the main thread
proc handle_http_event {increment_value max_progress_var} {
global progress_var

# Increment the progress variable by 10% of the maximum value
set progress_var [expr {$progress_var + $increment_value}]

# Ensure progress_var does not exceed max_progress_var
if {$progress_var >= $max_progress_var} {
set progress_var $max_progress_var
.progress.button configure -text "Done"
}

# Debugging output for the updated progress value
puts "Progress updated: $progress_var"

# Update the GUI to reflect the progress change
update_progress_bar
}

# Procedure to update the progress bar and label
proc update_progress_bar {} {
global progress_var max_progress_var percent_label

# Calculate the percentage done
set percent_done [expr {int(($progress_var / double($max_progress_var)) * 100)}]

# Update the label with the current percentage done
$percent_label configure -text "$percent_done%"

# Update the progress bar widget
.progress.progress configure -value $progress_var
}

# Main script to run the example
create_progress_bar "Progress Bar Example" 100

# Start the Tk event loop
tkwait window .progress

On 5/30/2024 1:50 PM, et99 wrote:

>                 # Pause briefly to simulate processing time
>                 after 100
>             }

Actually, this is not compute bound, but it should still work with some heavy compute, say a long for loop or computing a large Fibonacci number.

I didn't have the patience to try to get something that would trigger events inside the second thread. But since each thread is both inside a different system thread, and also uses a separate interpreter and event loop, I know from experience that this technique does work.

One caveat however, I would use a test inside the thread to not send a progress bar update for every single http event, rather, I'd only send one if it was a new integral value, like going from 10% to 11% rather than sending 10, 10.01, 10.02, ... since these small changes wouldn't likely show up in the progress bar gui anyway and might even cause a separate thread to start lagging.

also excuse my dyslexia, where I said send::thread :)

et99 <et99@rocketship1.me> wrote:
> # Send signal to main thread to update the progress variable
> thread::send -async \$main_thread_id \[list handle_http_event \$increment_value \$max_progress_var\]

Did it really write the above.

This will try to send to the thread id "$main_thread_id" (which, btw, is
not a thread ID that returns from thread::create) and will likely error
out for "too many arguments" before it errors out for "thread ID not
found" as thread::send takes at most three (ignoring option switches)
and the third one is a varname to save the result into when the script
exits.

I.e., it got the quoting all messed up.

On 5/30/2024 3:33 PM, Rich wrote:
> et99 <et99@rocketship1.me> wrote:
>> # Send signal to main thread to update the progress variable
>> thread::send -async \$main_thread_id \[list handle_http_event \$increment_value \$max_progress_var\]
>
> Did it really write the above.
>
> This will try to send to the thread id "$main_thread_id" (which, btw, is
> not a thread ID that returns from thread::create) and will likely error
> out for "too many arguments" before it errors out for "thread ID not
> found" as thread::send takes at most three (ignoring option switches)
> and the third one is a varname to save the result into when the script
> exits.
>
> I.e., it got the quoting all messed up.

If you look again, you will see that the script is defined using:

set script "....

"

It needs to define the thread script in quotes so it can substitute the variables from the main thread into the new thread script.

It's first attempt defined the thread::create with a script parameter in braces, not using a variable script. It only did that after I told it that the variables were not being substituted. I didn't tell it how to do that, it just figured it out on its own by using a variable named script.

I've just now asked it to stop using so many globals, so it re-wrote it using a namespace.

And I also asked it to replace the "after 100" with a compute bound for loop. When it did that, it got the quoting wrong again, but when I said the [expr] call also needed to be escaped, it fixed it, but it took a couple of tries. It's first attempt it used subst on the whole script and broke the rest of it.

You can copy/paste this code into a windows console (or probably run as a tclsh program from a file on linux, which I haven't tested though). In this code the new thread is compute bound yet the puts's and the progress bar update every 10% in the outer loop.

Once again, I only manually added the wm wi . and catch {console show}

And it still worked even after it told me my limit with the new chatGPT 4 was up and it reverted to using 3.5.

---------------------------

package require Tk
package require Thread
wm wi .
catch {console show}
namespace eval ProgressBar {
variable progress_var 0
variable max_progress_var 100
variable increment_value 10
variable percent_label ""
variable main_thread_id ""

# Procedure to create and initialize the progress bar
proc create {title max_value} {
variable progress_var
variable max_progress_var
variable increment_value
variable percent_label
variable main_thread_id

# Save the main thread ID
set main_thread_id [thread::id]

# Create the main window
set win [toplevel .progress]
wm title $win $title

# Create a label
label $win.label -text "Progress:"
pack $win.label -padx 10 -pady 5

# Initialize variables
set progress_var 0
set max_progress_var $max_value
set increment_value [expr {round($max_value * 0.10)}]

# Create a progress bar widget
ttk::progressbar $win.progress -variable progress_var -maximum $max_value -length 300
pack $win.progress -padx 10 -pady 10

# Create a label to show the percentage done
set percent_label [label $win.percent_label -text "0%"]
pack $percent_label -padx 10 -pady 5

# Button to start simulating fast HTTP events
button $win.button -text "Start Fast Events" -command [namespace code ::ProgressBar::simulate_http_events]
pack $win.button -padx 10 -pady 5
}

# Procedure to simulate very fast HTTP events
proc simulate_http_events {} {
variable max_progress_var
variable increment_value
variable main_thread_id

# Define the thread procedure script
set script "
proc thread_proc {main_thread_id increment_value max_progress_var} {
# Debugging output to indicate the thread is running
thread::send -async \$main_thread_id \[list puts \"Thread started\"\]
# Simulate fast HTTP events
for {set i 0} {\$i < 10} {incr i} {
# Debugging output for each iteration
thread::send -async \$main_thread_id \[list puts \"Thread iteration: \$i\"\]

# Send signal to main thread to update the progress variable
thread::send -async \$main_thread_id \[list ::ProgressBar::handle_http_event \$increment_value \$max_progress_var\]

# Simulate processing time with a compute-bound loop
set result 0
for {set j 0} {\$j < 10000000} {incr j} {
set result \[expr {\$result + \$j}\]
}
}
# Debugging output to indicate the thread has finished
thread::send -async \$main_thread_id \[list puts \"Thread finished\"\]
}
thread_proc $main_thread_id $increment_value $max_progress_var
"

# Create a new thread to handle fast HTTP events
set event_thread [thread::create $script]
}

# Procedure to handle HTTP event updates in the main thread
proc handle_http_event {increment_value max_progress_var} {
variable progress_var

# Increment the progress variable by 10% of the maximum value
set progress_var [expr {$progress_var + $increment_value}]

# Ensure progress_var does not exceed max_progress_var
if {$progress_var >= $max_progress_var} {
set progress_var $max_progress_var
.progress.button configure -text "Done"
}

# Debugging output for the updated progress value
puts "Progress updated: $progress_var"

# Update the GUI to reflect the progress change
update_progress_bar
}

# Procedure to update the progress bar and label
proc update_progress_bar {} {
variable progress_var
variable max_progress_var
variable percent_label

# Calculate the percentage done
set percent_done [expr {int(($progress_var / double($max_progress_var)) * 100)}]

# Update the label with the current percentage done
$percent_label configure -text "$percent_done%"

# Update the progress bar widget
.progress.progress configure -value $progress_var
}
}

# Main script to run the example
ProgressBar::create "Progress Bar Example" 100

# Start the Tk event loop
tkwait window .progress

et99 <et99@rocketship1.me> wrote:
> On 5/30/2024 3:33 PM, Rich wrote:
>> et99 <et99@rocketship1.me> wrote:
>>> # Send signal to main thread to update the progress variable
>>> thread::send -async \$main_thread_id \[list handle_http_event \$increment_value \$max_progress_var\]
>>
>> Did it really write the above.
>>
>> This will try to send to the thread id "$main_thread_id" (which, btw, is
>> not a thread ID that returns from thread::create) and will likely error
>> out for "too many arguments" before it errors out for "thread ID not
>> found" as thread::send takes at most three (ignoring option switches)
>> and the third one is a varname to save the result into when the script
>> exits.
>>
>> I.e., it got the quoting all messed up.
>
> If you look again, you will see that the script is defined using:
>
> set script "....
>
> "

Ah, yes, I did miss that little bit.

> I've just now asked it to stop using so many globals, so it re-wrote
> it using a namespace.

Yeah, I did notice, but did not comment on, the use of globals there.

It is interesting that it did not simply go the namespace route from
the start. Since if it could do it after you asked, it presumably
"knew how" when it wrote the version with the globals.

On 5/31/2024 9:27 AM, Rich wrote:
> et99 <et99@rocketship1.me> wrote:
>> On 5/30/2024 3:33 PM, Rich wrote:
>>> et99 <et99@rocketship1.me> wrote:
>>>> # Send signal to main thread to update the progress variable
>>>> thread::send -async \$main_thread_id \[list handle_http_event \$increment_value \$max_progress_var\]
>>>
>>> Did it really write the above.
>>>
>>> This will try to send to the thread id "$main_thread_id" (which, btw, is
>>> not a thread ID that returns from thread::create) and will likely error
>>> out for "too many arguments" before it errors out for "thread ID not
>>> found" as thread::send takes at most three (ignoring option switches)
>>> and the third one is a varname to save the result into when the script
>>> exits.
>>>
>>> I.e., it got the quoting all messed up.
>>
>> If you look again, you will see that the script is defined using:
>>
>> set script "....
>>
>> "
>
> Ah, yes, I did miss that little bit.
>
>> I've just now asked it to stop using so many globals, so it re-wrote
>> it using a namespace.
>
> Yeah, I did notice, but did not comment on, the use of globals there.
>
> It is interesting that it did not simply go the namespace route from
> the start. Since if it could do it after you asked, it presumably
> "knew how" when it wrote the version with the globals.
>

Yeah, it seems to like to use globals a lot. Perhaps internet training data uses globals more than namespaces. In its final code it's again using some globals, but I didn't ask it to fix that.

Final thoughts. I got chatGPT to write actual http code.

When it replaced the simulated http with the real deal it dumped the use of threads, which turned out to work just fine.

The secret sauce ended up being to update the progress bar only when it has changed by 1% or more and use update idletasks. I did have to request both of those. So, we're back to the (human) response from earlier in this thread.

One puzzle: I added some puts statements and I often see the http progress callback occurring repeatedly with the same bytes transferred. But since this doesn't increase the percent done, it doesn't try to update the progress. Maybe this is why Alexandru was having problems???

Also, for progress, it needs the expected size, so I asked it to write that code and it did but took 2 tries. I had to tell it how to read pages from https however, by just giving it code I've used before (with a test for windows vs. linux).

I then tested it out on some web pages, like wikipedia home page, and one longer page I found ("https://tkdocs.com/tutorial/onepage.html") but it doesn't always return an expected size value (returning 0 in that case). Today, it's working however. I tried to force it to read less by manually changing the http call to use -blocksize 1500 but that seems to be ignored for some reason.

One caveat, it keeps context, so when I made a big change request, it would sometimes mix up variables from earlier versions even though the code that had used them wasn't there any longer. In once case it has the same variable as a global but also in a namespace variable, which was pretty annoying.

Overall, I'm not totally sure it's much faster to use it, except when asking for some prototype code. Getting it to fix mistakes and make small changes was more time consuming than just doing it myself. However, you can paste in some modified code and it will continue from that point. And I was able to get it to translate to and from python code (it uses tkinter and the threading module when it was still using the thread package).

So, here's what I and it ended up with. The url it reads is in the code itself.

--------------------------

package require Tk
package require http

if { $::tcl_platform(platform) ne "windows" } {
package require tls
http::register https 443 [list ::tls::socket -ssl2 0 -ssl3 0 -tls1 1 -tls1.1 1 -tls1.2 1 ]
} else {
console show
package require twapi_crypto
http::register https 443 [list ::twapi::tls_socket]
}

wm withdraw .

namespace eval ProgressBar {
variable progress_var 0
variable max_progress_var 0
variable increment_value 1
variable percent_label ""
variable last_percent -1 ;# Initialize to -1 to ensure initial update

# Procedure to create and initialize the progress bar
proc create {title max_value} {
variable progress_var
variable max_progress_var
variable increment_value
variable percent_label

# Create the main window
set win [toplevel .progress]
wm title $win $title

# Create a label
label $win.label -text "Progress:"
pack $win.label -padx 10 -pady 5

# Initialize variables
set progress_var 0
set max_progress_var $max_value
set increment_value 1

# Create a progress bar widget
ttk::progressbar $win.progress -maximum 100 -length 300
pack $win.progress -padx 10 -pady 10

# Create a label to show the percentage done
set percent_label [label $win.percent_label -text "0%"]
pack $percent_label -padx 10 -pady 5

# Button to start simulating fast HTTP events
button $win.button -text "Start Fast Events" -command [namespace code ::ProgressBar::start_or_reset]
pack $win.button -padx 10 -pady 5

}

# Procedure to start or reset the progress
proc start_or_reset {} {
variable progress_var
variable max_progress_var
puts start
if {$progress_var >= $max_progress_var} {
# Reset progress
set progress_var 0
.progress.button configure -text "Start Fast Events"
}

# Start the HTTP request with progress tracking
start_http_request_with_progress_tracking
}

# Function to initialize progress tracking
proc init_progress_tracking {expected_total} {
global total_size segment_size
set total_size $expected_total
set segment_size [expr {$total_size / 100}]
}

# Callback function to handle HTTP progress
proc httpProgress {token total_size bytes_transferred} {
global progress_var max_progress_var increment_value last_percent

# Initialize last_percent if not already initialized
if {! [info exists last_percent]} {
set last_percent -1
}

# Initialize progress tracking if not already initialized
if {! [info exists total_size]} {
ProgressBar::init_progress_tracking $total_size
}

# Calculate the progress if the total size is available and not zero
if {$total_size > 0} {
set progress [expr {int(($bytes_transferred / double($total_size)) * 100)}]
} else {
# Handle the case where total size is not available
set progress [expr {int(($bytes_transferred / double($max_progress_var)) * 100)}]

}

# Update the progress bar only when the percentage changes
if {$progress != $last_percent} {
puts "last_percent= |$last_percent| progress= |$progress| "
set last_percent $progress

ProgressBar::update_progress_bar $progress
# after 100 ;# manually added so I can see visually the progress bar change
}

# Increment the progress variable
set progress_var [expr {$progress_var + $increment_value}]

# Ensure progress_var does not exceed max_progress_var
if {$progress_var >= $max_progress_var} {
set progress_var $max_progress_var
.progress.button configure -text "Done"
}
}

# Function to update the progress bar and label
proc update_progress_bar {percent_done} {
variable max_progress_var

# Update the label with the current percentage done
.progress.percent_label configure -text "$percent_done%"

# Update the progress bar widget
.progress.progress configure -value $percent_done
update idletasks
}

# Main procedure to start the HTTP request with progress tracking
proc start_http_request_with_progress_tracking {} {
global progress_var max_progress_var increment_value
puts inside-start
# Initialize progress tracking variables
set progress_var 0
set max_progress_var 130595 ;# Set the maximum progress value
set increment_value 100 ;# Increment value for each progress update

# Start the HTTP request with progress callback
set url "https://en.wikipedia.org/wiki/Main_Page"
set url "https://tkdocs.com/tutorial/onepage.html"
set token [::http::geturl $url -blocksize 1500 -progress [namespace code {httpProgressCallback}]]
puts "token= |$token| "
# Wait for the HTTP request to complete
::http::wait $token
puts done
}

# Callback function to handle HTTP progress
proc httpProgressCallback {token total_size bytes_transferred} {
puts "token= |$token| total_size= |$total_size| bytes_transferred= |$bytes_transferred| "

# Call httpProgress with the provided arguments
httpProgress $token $total_size $bytes_transferred
}

# Callback function to handle HTTP response
proc httpResponseCallback {token} {
global total_size
set headers [::http::meta $token]
set content_length [dict get $headers "content-length"]
set total_size [expr { $content_length != "" ? $content_length : -1 }]

# Initialize progress tracking with the total size
ProgressBar::init_progress_tracking $total_size
}
}

On 5/31/2024 12:05 PM, et99 wrote:
-snip-

I just noticed that it's not actually calling the code it wrote to get the total expected size. It appears that the progress callback is actually doing that and passing it in. It seems to sometimes end up with fossil code that it doesn't clean up.

et99 <et99@rocketship1.me> wrote:
> On 5/31/2024 9:27 AM, Rich wrote:
>> Yeah, I did notice, but did not comment on, the use of globals
>> there.
>>
>> It is interesting that it did not simply go the namespace route from
>> the start. Since if it could do it after you asked, it presumably
>> "knew how" when it wrote the version with the globals.
>
> Yeah, it seems to like to use globals a lot. Perhaps internet
> training data uses globals more than namespaces. In its final code
> it's again using some globals, but I didn't ask it to fix that.

This is a reasonable assumption -- unfortunately we don't know exactly
what it was tained upon so we can only guess why it might prefer
globals.

> One puzzle: I added some puts statements and I often see the http
> progress callback occurring repeatedly with the same bytes
> transferred.

Not surprising. When asked to run the callback the module reads in
chunks of -blocksize, so each callback will be after -blocksize bytes
of the body of the http request have arrived.

Note from the http manpage:

-blocksize size
The block size used when reading the URL. At most size bytes are
read at once. After each block, a call to the -progress callback
is made (if that option is specified).

And in at least the 2.9.5 version that ships with Tcl 8.6.12
-blocksize defaults to 8192 bytes if not specified by the caller.

> But since this doesn't increase the percent done, it doesn't try to
> update the progress. Maybe this is why Alexandru was having
> problems???

Most likely it was the difference in speed between "local lan" and
"VPN". The VPN added just enough overhead that the [update] could
update the progress bar and exit without retriggering the [read] inside
the http modulee. Running on the local lan the bytes were arriving
quickly enough that the [update] inside the callback caused the read
event to retrigger for the http module, while still in the [update],
resulting in the recursive calls to the event loop.

> I then tested it out on some web pages, like wikipedia home page, and
> one longer page I found ("https://tkdocs.com/tutorial/onepage.html")
> but it doesn't always return an expected size value (returning 0 in
> that case).

Do note that the "expected size" for a http transaction depends upon
the server including the "size" header in the HTTP headers, and that
"size" is optional (to allow for situations where the server does not
know, at the start of sending the data back to the client, just how
many bytes will eventually be generated. So whether you get a
"expected size" value depends upon whether the server sends one.

On 5/31/2024 12:32 PM, Rich wrote:
> et99 <et99@rocketship1.me> wrote:
>> On 5/31/2024 9:27 AM, Rich wrote:

>> One puzzle: I added some puts statements and I often see the http
>> progress callback occurring repeatedly with the same bytes
>> transferred.
>
> Not surprising. When asked to run the callback the module reads in
> chunks of -blocksize, so each callback will be after -blocksize bytes
> of the body of the http request have arrived.
>
> Note from the http manpage:
>
> -blocksize size
> The block size used when reading the URL. At most size bytes are
> read at once. After each block, a call to the -progress callback
> is made (if that option is specified).
>
> And in at least the 2.9.5 version that ships with Tcl 8.6.12
> -blocksize defaults to 8192 bytes if not specified by the caller.
>

That makes sense, but I would expect it to call the callback with a new value for bytes transferred, not the same value.

And I did use a -blocksize 1500 and I still see values like, 16384 repeated 5 times, then 32768 (this is on windows) so that would seem to indicate it's reading in blocks of 2*8192 (same with 8.6.9 and 8.6.13). Also, it even did 4 callbacks at the start with a value of 0.

et99 <et99@rocketship1.me> wrote:
> On 5/31/2024 12:32 PM, Rich wrote:
>> et99 <et99@rocketship1.me> wrote:
>>> On 5/31/2024 9:27 AM, Rich wrote:
>
>>> One puzzle: I added some puts statements and I often see the http
>>> progress callback occurring repeatedly with the same bytes
>>> transferred.
>>
>> Not surprising. When asked to run the callback the module reads in
>> chunks of -blocksize, so each callback will be after -blocksize
>> bytes of the body of the http request have arrived.
>>
>> Note from the http manpage:
>>
>> -blocksize size
>> The block size used when reading the URL. At most size bytes
>> are read at once. After each block, a call to the -progress
>> callback is made (if that option is specified).
>>
>> And in at least the 2.9.5 version that ships with Tcl 8.6.12
>> -blocksize defaults to 8192 bytes if not specified by the caller.
>>
>
> That makes sense, but I would expect it to call the callback with a
> new value for bytes transferred, not the same value.

It is not reporting "total bytes so far" it is reporting "bytes for
this callback". The callback has to take care of totalling things up
if that is what it wants.

> And I did use a -blocksize 1500 and I still see values like, 16384
> repeated 5 times, then 32768 (this is on windows) so that would seem
> to indicate it's reading in blocks of 2*8192 (same with 8.6.9 and
> 8.6.13). Also, it even did 4 callbacks at the start with a value of
> 0.

Unsure on that. Probably it is calling the callback per event loop
cycle, and if the read event fires for some reason, but there are no
bytes, you'd see a zero. Or else those are for the headers and it just
reports zero for headers (as headers are not body bytes).

Why the size is larger than what you set for blocksize I can't say.

On 5/31/2024 2:14 PM, Rich wrote:
> et99 <et99@r...........> wrote:
>> On 5/31/2024 12:32 PM, Rich wrote:

> It is not reporting "total bytes so far" it is reporting "bytes for
> this callback". The callback has to take care of totalling things up
> if that is what it wants.
>

Here's the manpage on -progress

-progress callback
The callback is made after each transfer of data from the URL. The callback gets three additional arguments: the token from ::http::geturl, the expected total size of the contents from the Content-Length meta-data, and the *current number of bytes transferred so far*. The expected total size may be unknown, in which case zero is passed to the callback.

(emphasis added)

>> And I did use a -blocksize 1500 and I still see values like, 16384
>> repeated 5 times, then 32768 (this is on windows) so that would seem
>> to indicate it's reading in blocks of 2*8192 (same with 8.6.9 and
>> 8.6.13). Also, it even did 4 callbacks at the start with a value of
>> 0.
>
> Unsure on that. Probably it is calling the callback per event loop
> cycle, and if the read event fires for some reason, but there are no
> bytes, you'd see a zero. Or else those are for the headers and it just
> reports zero for headers (as headers are not body bytes).
>
> Why the size is larger than what you set for blocksize I can't say.

Here's the output of the program, it outputs last_percent... only when it updates the progress bar, so only when the bytes_transferred has increased:

start
inside-start
token= |::http::1| total_size= |170427| bytes_transferred= |0|
last_percent= |-1| progress= |0|
token= |::http::1| total_size= |170427| bytes_transferred= |0|
token= |::http::1| total_size= |170427| bytes_transferred= |0|
token= |::http::1| total_size= |170427| bytes_transferred= |0|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
last_percent= |0| progress= |9|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |16384|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
last_percent= |9| progress= |19|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
token= |::http::1| total_size= |170427| bytes_transferred= |32768|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
last_percent= |19| progress= |28|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |49152|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
last_percent= |28| progress= |38|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
token= |::http::1| total_size= |170427| bytes_transferred= |65536|
token= |::http::1| total_size= |170427| bytes_transferred= |81920|
last_percent= |38| progress= |48|
token= |::http::1| total_size= |170427| bytes_transferred= |81920|
token= |::http::1| total_size= |170427| bytes_transferred= |81920|
token= |::http::1| total_size= |170427| bytes_transferred= |81920|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
last_percent= |48| progress= |57|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
token= |::http::1| total_size= |170427| bytes_transferred= |98304|
token= |::http::1| total_size= |170427| bytes_transferred= |114688|
last_percent= |57| progress= |67|
token= |::http::1| total_size= |170427| bytes_transferred= |114688|
token= |::http::1| total_size= |170427| bytes_transferred= |114688|
token= |::http::1| total_size= |170427| bytes_transferred= |114688|
token= |::http::1| total_size= |170427| bytes_transferred= |114688|
token= |::http::1| total_size= |170427| bytes_transferred= |131072|
last_percent= |67| progress= |76|
token= |::http::1| total_size= |170427| bytes_transferred= |131072|
token= |::http::1| total_size= |170427| bytes_transferred= |131072|
token= |::http::1| total_size= |170427| bytes_transferred= |131072|
token= |::http::1| total_size= |170427| bytes_transferred= |131072|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
last_percent= |76| progress= |86|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
token= |::http::1| total_size= |170427| bytes_transferred= |147456|
token= |::http::1| total_size= |170427| bytes_transferred= |163840|
last_percent= |86| progress= |96|
token= |::http::1| total_size= |170427| bytes_transferred= |163840|
token= |::http::1| total_size= |170427| bytes_transferred= |170427|
last_percent= |96| progress= |100|
token= |::http::1|
done

et99 <et99@rocketship1.me> wrote:
> On 5/31/2024 2:14 PM, Rich wrote:
>> et99 <et99@r...........> wrote:
>>> On 5/31/2024 12:32 PM, Rich wrote:
>
>> It is not reporting "total bytes so far" it is reporting "bytes for
>> this callback". The callback has to take care of totalling things
>> up if that is what it wants.
>>
>
> Here's the manpage on -progress
>
> -progress callback
> The callback is made after each transfer of data from the URL.
> The callback gets three additional arguments: the token from
> ::http::geturl, the expected total size of the contents from the
> Content-Length meta-data, and the *current number of bytes
> transferred so far*. The expected total size may be unknown, in
> which case zero is passed to the callback.
>
> (emphasis added)

I stand corrected. So no idea why the callback is being called when no
additional bytes have been transferred (which would be how one would
get the same size in two or more callbacks).

>>> And I did use a -blocksize 1500 and I still see values like, 16384
>>> repeated 5 times, then 32768 (this is on windows) so that would seem
>>> to indicate it's reading in blocks of 2*8192 (same with 8.6.9 and
>>> 8.6.13). Also, it even did 4 callbacks at the start with a value of
>>> 0.
>>
>> Unsure on that. Probably it is calling the callback per event loop
>> cycle, and if the read event fires for some reason, but there are no
>> bytes, you'd see a zero. Or else those are for the headers and it just
>> reports zero for headers (as headers are not body bytes).
>>
>> Why the size is larger than what you set for blocksize I can't say.
>
> Here's the output of the program, it outputs last_percent... only
> when it updates the progress bar, so only when the bytes_transferred
> has increased:
>
> start
> inside-start
> token= |::http::1| total_size= |170427| bytes_transferred= |0|
> last_percent= |-1| progress= |0|
> token= |::http::1| total_size= |170427| bytes_transferred= |0|
> token= |::http::1| total_size= |170427| bytes_transferred= |0|
> token= |::http::1| total_size= |170427| bytes_transferred= |0|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> last_percent= |0| progress= |9|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |16384|
> token= |::http::1| total_size= |170427| bytes_transferred= |32768|

Indeed, that is unexpected.

I get the same result on Linux:

start
inside-start
token= |::http::1| total_size= |170427| bytes_transferred= |932|
last_percent= |-1| progress= |0|
token= |::http::1| total_size= |170427| bytes_transferred= |932|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
last_percent= |0| progress= |9|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |15926|
token= |::http::1| total_size= |170427| bytes_transferred= |32310|
last_percent= |9| progress= |18|

Just with slightly different amount of data transferred each callback.

On 5/31/2024 10:01 PM, Rich wrote:
> et99 <et99@rocketship1.me> wrote:
>> On 5/31/2024 2:14 PM, Rich wrote:
>>> et99 <et99@r...........> wrote:
>>>> On 5/31/2024 12:32 PM, Rich wrote:
>>
>
> Indeed, that is unexpected.
>
> I get the same result on Linux:
>

> Just with slightly different amount of data transferred each callback.

Thanks for verifying that.

I have discovered one thing, if I add a delay [after 300] inside the callback, the repeats are gone.

It thus seems as though the implementation of the progress callback is via a fixed timer schedule and reports the values as they are in the internal counters, without checking if anything has changed.

I've sent in a ticket.