[I am the author of the arxiv paper 0608238v2.pdf, from 1986.
I have just started reading sci.stat.math. I will respond in this post
to all of the posts in this thread that exist right now, consolidating
them; I will respond to the thread as long as it makes sense to do so. I
will not read the unrelated alt.usage.english, and after this post will
not include it. I am cross-posting to sci.physics.relativity.]

On 2/27/23 3:00 PM, Rich Ulrich wrote:
> [Roberts' new] analysis itself occupies about two pages in Section
> IV of this article:
> https://arxiv.org/vc/physics/papers/0608/0608238v2.pdf

That is the paper I wrote, back in 1986.

> Modern statistical analyses and design sophistication for statistics
> were barely being born in 1933, when the Miller experiment was
> published. [...]

Yes. I mentioned that in the paper. Worse than lack of statistical
errorbars is Miller's lack of knowledge of digital signal processing --
his analysis is essentially a comb filter that concentrates his
systematic error into the DFT bin corresponding to a real signal --
that's a disaster, and explains why his data reduction yields data that
look like a sinusoid with period 1/2 turn. In short, this is every
experimenter's nightmare: he was unknowingly looking at statistically
insignificant patterns in his systematic drift that mimicked the
appearance of a real signal.

See sections II and III of the paper.

> Also, 'messy data' (with big sources of random error) remains a
> problem with solutions that are mainly ad-hoc (such as, when Roberts
> offers analyses that drop large fractions of the data).

I did not "drop large fractions of the data", except that I analyzed
only 67 of his data runs, out of more than 1,000 runs. As my analysis
requires a computer, it is necessary to type the data from copies of
Miller's data sheets into the computer. I do not apologize for doing
that for only a small fraction of the runs (I had help from Mr. Deen).
The 67 runs in section IV of the paper are every run that I had.

> Roberts shows me that these data are so messy that it is hard to
> imagine Miller retrieveing a tiny signal from the noise, if Miller
> did nothing more than remove linear trends from each cycle.

Yes. See figures 2,3,4 of the paper. A glance at Fig. 2 shows how
terrible the drift actually is (almost 6 fringes over 20 turns, more
than 50 times larger than the "signal" Miller plotted in Fig. 1). The
fact that the dots do not lie on the lines of Fig. 3 shows how
inadequate it is to assume a linear drift, by an amount as much as
10 times larger than the "signal" he plotted.

Had Miller displayed his actual data plots, like my Fig. 2, or the
nonlinearities as in my Fig. 3, nobody would have believed he could
extract a signal with a peak-to-peak amplitude <0.1 fringe. Both of
those are well within his capabilities.

> I would want to know how the DEVICE made all those errors possible,

It is drifting, often by large amounts -- so large that in most runs
Miller actually changed the interferometer alignment DURING THE RUN by
adding weights to one of the arms (three times in the run of Fig. 1).
Even so, there are often jumps between adjacent data points of a whole
fringe or more -- that is unphysical, and can only be due to an
instrumentation instability.

Modern interferometers are ENORMOUSLY more stable. In the precision
optics lab I manage, we have a Michelson interferometer that is ~ 10,000
times more stable than Miller's. We use it to stabilize lasers, not
search for an aether. That stability includes a lack of 12-hour
variations, with a sensitivity of ~ 0.00002 fringe (~ 10,000 times
better than Miller's).

> If you are wondering about how he fit his model, I can say a little
> bit. The usual fitting in clinical research (my area) is with
> least-squares multiple regression, which minimizes the squared
> residuals of a fit. The main alternative is Maximum Likelihood,
> which finds the maximum likelihood from a Likelihood equation. That
> is evaluated by chi-squared ( chisquared= -2*log(likelihood) ).
> Roberts seems to be using some version of that, though I didn't yet
> figure out what he is fitting.

See Section IV of the paper. As described, the analysis starts by
modeling the data as
data = signal(orientation) + systematic(time)
The challenge is to separate these two components. By taking advantage
of the 180-degree symmetry of the instrument, only 8 orientations are
used. Since signal(orientation) is the same for every 1/2 turn, by
subtracting the data of the first 1/2-turn from the data for every 1/2
turn, signal(orientation) is canceled and the result contains just
systematic(time), with each orientation individually set to 0 at the
first point (of 40). The time dependence of each orientation is
preserved. Here "time" is represented by data points taken successively
at each of 16 markers for each of 20 turns, so there are 16*20=320
"time" points; my plots are labeled "Turn" (not "time").

Once the systematic has been isolated for each orientation (see Fig.
10), the idea is to restore the time dependence of the systematic and
then subtract it from the data. Because the first 1/2 turn was
subtracted everywhere, each of the 8 orientations starts at 0. So to put
them together into a single time sequence I introduced 8 parameters,
each representing the systematic value for one orientation in the first
1/2 turn. Because the ChiSq is a sum of differences, it is necessary to
fix the overall normalization, which I did by holding the parameter for
markers 1 and 9 fixed at 0. So the fit varies 7 parameters with the goal
of making the time series as smooth as possible. The ChiSq is the sum of
319 terms corresponding to the differences between successive points of
the time series for the systematic (a difference for each dot in Fig. 10
except the first). Note each entry is subtracting values for two
successive orientations, because that is how the data were collected;
this is clearly a measure of the smoothness of the overall time
sequence. The errorbar for computing the ChiSq was set at 0.1 fringe,
because that is the quantization of the data; similarly the parameters
were quantized at 0.1 fringe. Conventional fitting programs don't work
with quantized parameters (they need derivatives), so I just performed
an exhaustive search of sets of the 7 parameters, looking for minimum ChiSq.

Note I did NOT do the simple and obvious thing: use the data for the
first 1/2 turn as the values of the parameters. That would reintroduce
signal(orientation) and make the analysis invalid.

Once the full time sequence of the systematic drift has been determined,
it is subtracted from the raw data to obtain signal(orientation). For
most of the runs (53 out of 67, closed circles in Fig. 11), the
systematic model reproduces the data exactly. The other 14 runs exhibit
gross instability (see section IV of the paper).

> I thought it /was/ appropriate that he took the consecutive
> differences as the main unit of analysis, given how much noise there
> was in general. From what I understood of the apparatus, those are
> the numbers that are apt to be somewhat usable.
>
> Ending up with a chi-squared value of around 300 for around 300 d.f.
> is appropriate for showing a suitably fitted model -- the expected
> value of X2 by chance for large d.f. is the d.f. A value much
> larger indicates poor fit; much smaller indicates over-fit.

Yes. In this case it means my guess of 0.1 fringe for the errorbars was
appropriate.

David Jones wrote:
> I have heard some non-statistical experts in other fields just using
> "chi-squared" to mean a sum of squared errors.

I used the term as it is commonly used in physics. It is a sum of
squared differences each divided by its squared errorbar. Having no
actual errorbars, I approximated them by using a constant 0.1 fringe,
which is the quantization Miller used in recording the data.

> My guess is that some of the stuff in this paper is throwing-out
> some information about variability in whatever "errors" are here.

Within each run, no data were "thrown out"; from the set of 67 runs I
had, no runs were "thrown out". But criticism about using just 67 runs
out of >1,000 is valid. But realistically, since these 67 runs show no
significant signal, and display such enormous drift of the instrument,
does anybody really expect the other runs to behave differently?

> If this were a simple time series, one mainstream approach from
> "time-series analysis" would be to present a spectral analysis of a
> detrended and prefiltered version of the complete timeseries, to try
> to highlight any remaining periodicities.

Fig. 6 is a DFT of the data considered as a single time series 320
samples long, for the run in Fig. 1. Here "time" is sample number, 0 to 320.

> There would seem to be a possibility of extending this to remove
> other systematic effects.

What "other systematic effects"? -- all of them are contained in
Miller's data, which were used to model systematic(time).

> I think the key point here is to try to separate-out any isolated
> frequencies that may be of interest, rather than to average across a
> range of neighbouring frequencies, as may be going on in this
> paper.

The only frequency of interest is that corresponding to 1/2 turn, where
any real signal would be. The norm of that amplitude is what the paper
presents in Fig. 11.

> To go any further in understanding this one would need to have a
> mathematical description of whatever model is being used for the full
> data-set, together with a proper description of what the various
> parameters and error-terms are supposed to mean.

Read the paper, and my description above.

IMHO further analysis is not worth the effort -- Miller's data are so
bad that further analysis is useless.

Similar experiments with much more stable interferometers have detected
no significant signal.

> The date of the paper is not clear.

Arxiv says it was last revised 15 Oct 2006; the initial submission year
and month are enshrined in the first four digits of the filename.

Anton Shepelev wrote:
> there are no time readings in Miller's data.

Yes, but that doesn't matter, as time is not relevant; orientation is
relevant, and that is represented by successive data points, 16
orientations for each of 20 turns.

> knowing the importance of this seminal experiment [...]

Miller's experiment is "seminal" only to cranks and people who don't
understand basic experimental technique. The interferometer is so very
unstable that his measurements are not worth anything -- Note that
Miller never presented plots of his data (as I did in Fig. 2). Had he
displayed such plots, nobody would have believed he could extract a
signal with a peak-to-peak amplitude < 0.1 fringe. Ditto for the
nonlinearity shown in Fig. 3.

> [I] realy should have found the time, resources, and help to
> digitise the entire data.

Where do you suppose that would come from? Remember that Rev. Mod. Phys.
would not even publish the paper, stating that the subject is too old
and no longer of interest. No sensible funding agency would support
further research on this. IMHO further analysis is not worthwhile, as
his instrument is so very unstable.

In the precision optics lab I manage, we have a Michelson interferometer
that is ~ 10,000 times more stable than Miller's. We use it to stabilize
lasers, not search for an aether. That stability includes a lack of
12-hour variations, with a sensitivity of ~ 0.00002 fringe.

> [further analysis is] impossible without Miller's original data

Miller's original data sheets are available from the CWRU archives. They
charge a nominal fee for making copies. IIRC there are > 1,000 data
sheets. Transcribing them into computer-readable form is a daunting
task, and as I have said before, IMHO it is simply not worthwhile.

> Exactly, and I bet it is symbolic parametrised funtions that you
> fit, and that your models include the random error (noise) with
> perhaps assumtions about its distribution.

I don't know what you are trying to say here, nor who "you" is.

> No so with Roberts's model, which is neither symblic nor has noise as
> an explicit term!

Hmmm. I don't know what "symblic" means. But yes, my model has no
explicit noise term because it is piecing together the systematic error
from the data with the first 1/2 turn subtracted; any noise is already
in that data. Virtually all of the variation is a systematic drift, not
noise, and I made no attempt to separate them.

> They /are/ usable in that they still contain the supposed signal and
> less random noise (because of "multisampling"). But you will be
> surprised if you look at what that does to the systematic error!

Hmmm. The key idea was to subtract the first 1/2 turn from every half
turn, to remove the signal(orientation), leaving just systematic(time),
with each orientation individually zeroed in the first point (of 40).

> My complaint, however, is about the model that he fitted, and the
> way he did it -- by enumerating the combinations of the seven free
> parameters by sheer brute force.

Hmmm. With quantized data and quantized parameters, no conventional
fitting program will work, as they all need derivatives; enumerating the
parameter values was the only way I knew how to find the best set of
parameters. As the paper says, it typically took about 3 minutes per
run (on a now 40-year-old laptop), so this brute force approach was
plenty good enough.

Note the quantization was imposed by Miller's method of taking data, not
anything I did.

> Roberts jumped smack dab into the jaws of the curse of
> dimensionality where I think nothing called for it!

I have no idea of what you mean.

> He even had to "fold" the raw data in two -- to halve the degrees of
> freedom. I wonder what he would say to applying that technique to an
> experiment with 360 measurements per cycle!

a) "Folding" the data is due to the symmetry of the instrument and
is solidly justified on physics grounds.
b) I did apply the analysis to an experiment with 320 measurements
per run.
c) With this algorithm, the main driver for computer time is the
number of parameters, not the number of measurements [#].
The 7 parameters are due to the instrument and Miller's
method of data taking, not anything I did.
d) if you meant 360 orientations, that would indeed be infeasible
to analyze with this algorithm, even with supercomputer support.
But to get data like that would require a completely new
instrument, and it would be silly to not make it as stable
as modern technology permits, so a better algorithm could
surely be devised.

[#] IIRC in the enumeration I used the ten most
likely values for each parameter, so the computer
time for N parameters and K measurements is
roughly proportional to (10^N)*K.

> Miller, considering the level of statistical science in 1933, did a
> top-notch job. Both his graphs and results of mechanical harmonic
> analysis[1] show a dominance of the second harmonic in the signal,
> albeit at a much lower magnitude that initially expected.

See section III of my paper for why the second harmonic dominates -- his
analysis algorithm concentrates his systematic drift into the lowest DFT
bin, which "just happens" to be the second harmonic bin where any real
signal would be. Had Miller displayed plots of his raw data, like my
Fig. 2, nobody would have believed he could extract such a small
"signal" from such messy data. Ditto for the nonlinearities shown in
Fig. 3. Both plots are well within his capabilities.

Go look at my Fig. 2 -- do you seriously think you can extract a
sinewave signal with amplitude ~ 0.1 fringe from that data? Miller
fooled himself into thinking he could, but today you are not constrained
by the lack of knowledge and understanding that he had back in 1933.

> Actually, the sequences of consequtive interferometer "runs" may be
> considered as uninterrupted time series, with the reservation that
> the data has no time readings, because the experimenters did not
> intend it for such analysis.

Not true, as Miller re-aligned the instrument between runs. Indeed he
often realigned the instrument within runs. The need for such frequent
re-alighments indicates how very unstable his instrument is. (The
Michelson interferometer in our lab is realigned every few months, not
every few minutes as Miller's required.)

> The problem of separating the systematic error from the signal is
> quite hard and, in my opinion, requires an accurately consturcted
> model, which Roberts seems to lack.

Go back and read my paper. I developed an excellent model of the
systematic drift for each run I analyzed.

> Robert's model is:
>
> singnal(orientation) + system_error(time)
>
> but he seems to be confused about what he means by time. At one
> point he says it is the number of the interferometer revolution, at
> another he seems to imply that the sequence of sixteen readings
> /during/ a revolution is also time.

_I_ am not confused, but perhaps my description is not as clear as it
could be. As I said in the paper and above, "time" is represented by
successive data points. I used units of turns, with each successive
marker incrementing by 0.0625 turn; each run has "time" from 0 to 20.0
turns. (Miller's and my "turn" = your "revolution".)

> But then, this kind of time includes orientation, because,
> naturally, the device rotates in time. I therefore fail to comprehend
> how this model gurrantees that the singal is not misinterpreted as
> part of systematic error.

The key point is that signal(orientation) is the same for every 1/2
nturn. So by subtracting the data from the first 1/2 turn from each 1/2
turn, signal(orientation) is canceled throughout and the result contains
just systematic(time), with each orientation individually zeroed for the
first point (of 40). See Fig. 10, noting that each orientation has its
own 0 along the vertical axis.

> Also -- where is random error in the model?

It is contained in Miller's data. I made no attempt to distinguish a
systematic drift from random noise or error; in this algorithm there's
no need to do so.

> All in all, I am utterly confused by Roberts's model from the start.

Perhaps a discussion can resolve your confusion.

BTW I still have these 67 runs on disk. If anyone wants them, just ask.
I am surprised that the analysis program source is not also there, but
it isn't, and I doubt it is still accessible. IIRC it was about 10 pages
of Java.

Tom Roberts