Rocksolid Light

News from da outaworlds

mail  files  register  groups  login

Message-ID:  

BOFH excuse #131: telnet: Unable to connect to remote host: Connection refused

comp / comp.lang.python / Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

SubjectAuthor
* Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Ke2QdxY4RzWzUUiLuE
`* Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from KeKeith Thompson
 `- Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Ke2QdxY4RzWzUUiLuE

1
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
From: 2QdxY4RzWzUUiLuE@potatochowder.com
Newsgroups: comp.lang.python
Date: Tue, 1 Oct 2024 00:06 UTC
References: 1 2 3 4 5 6 7 8 9 10 11
Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!fu-berlin.de!uni-berlin.de!not-for-mail
From: 2QdxY4RzWzUUiLuE@potatochowder.com
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
GB) from Kenna API
Date: Mon, 30 Sep 2024 20:06:57 -0400
Lines: 42
Message-ID: <mailman.18.1727741223.3018.python-list@python.org>
References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
<CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
<CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
<CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
<4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
<CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
<ZvsjI-E8Qtz4rCeL@anomaly> <4XHbxS5jl4znVGD@mail.python.org>
<CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
<Zvs9IZ5zxaosKcln@anomaly>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: news.uni-berlin.de oZX7X4MGVv/x6EsI3E5Y1Ql7labU9XQaAML8kabzc9Pg==
Cancel-Lock: sha1:12eNQcVpIUvOGadVbM/Dwhw93MU= sha256:8j85O3ULqmov18aCJpcZc0L3rzVqr+Y4jAM+axzxMMg=
Return-Path: <2QdxY4RzWzUUiLuE@potatochowder.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=potatochowder.com header.i=@potatochowder.com
header.b=W7j3AuLi; dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.011
X-Spam-Evidence: '*H*': 0.98; '*S*': 0.00; 'looks': 0.02; 'variable':
0.05; 'subject:API': 0.07; 'angelico': 0.09; 'characters,': 0.09;
'dan': 0.09; 'meant': 0.09; 'received:78': 0.09; '2024': 0.16;
'anyway.': 0.16; 'base.': 0.16; 'decimal': 0.16;
'from:addr:2qdxy4rzwzuuilue': 0.16; 'from:addr:potatochowder.com':
0.16; 'iirc,': 0.16; 'integer': 0.16; 'literals': 0.16; 'odd':
0.16; 'recall': 0.16; 'received:136.243': 0.16; 'received:172.58':
0.16; 'received:78.46': 0.16; 'received:78.46.172': 0.16;
'received:www458.your-server.de': 0.16; 'received:your-server.de':
0.16; 'wrote:': 0.16; 'subject:Help': 0.17; 'grant': 0.17; '16,':
0.19; 'tue,': 0.19; 'to:addr:python-list': 0.20; 'input': 0.21;
'written': 0.22; 'received:de': 0.23; 'anything': 0.25; 'chris':
0.28; 'think': 0.32; "doesn't": 0.32; '(as': 0.32; 'end.': 0.32;
'python-list': 0.32; 'received:136': 0.32; 'but': 0.32; "i'm":
0.33; 'subject:for': 0.33; 'header:In-Reply-To:1': 0.34; 'yes,':
0.35; 'presented': 0.37; 'subject:from': 0.37; "it's": 0.37;
'hard': 0.37; 'put': 0.38; 'read': 0.38; 'two': 0.39; 'least':
0.39; 'still': 0.40; 'base': 0.40; 'both': 0.40; 'tell': 0.60;
'10,': 0.61; 'above': 0.62; 'lower': 0.62; 'between': 0.63;
'leading': 0.63; 'our': 0.64; 'clear': 0.64; 'look': 0.65;
'numbers': 0.67; 'right': 0.68; 'matter': 0.68; 'skip:# 10': 0.68;
'interpreted': 0.69; 'knowing': 0.71; 'subject:Data': 0.71;
'global': 0.73; 'little': 0.73; 'significant': 0.78; 'admit':
0.84; 'bind': 0.84; 'handled': 0.84; 'identifies': 0.84; 'six,':
0.84; 'subject: \n ': 0.84
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=potatochowder.com; s=default2305; h=In-Reply-To:Content-Type:MIME-Version:
References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc:
Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:
Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID;
bh=M5OrN57pRSm7isZdVzSbBYsl+jpXgR8mfmw4JEnhFEA=; b=W7j3AuLiQWTKbUoTYsaBvVvwiB
yvPF7ofcPZ1y7qMU7R9bNY3jTEJcLnzz4m2Ik+N5B3TbW7aicJ7PGUBsq8jiYsy7ntGFPErpb02Po
D0E3JDAlpXu+vjccbSYEtGUen3vcOUnjPcCFGdNSCRqvMk1teCPNB/yWkyDl55MUyZwPcPbAATs/U
7J3hJwoAhPA484kpZtspUTeZxbQwIFiLZKj6dfaoQ/hywmRf+ApTQaieuCQV8fRKtM4Q/ua/s6X+q
/A3Zw+Wgfz9/PJ/kD8S5tl6KutE116hm0BmNGUaeI2mWCgudqAesFj6Ktib0qfGS5gwNpNPCW/9fX
YK6aJ1ow==;
Mail-Followup-To: python-list@python.org
Content-Disposition: inline
In-Reply-To: <CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
X-Authenticated-Sender: 2QdxY4RzWzUUiLuE@potatochowder.com
X-Virus-Scanned: Clear (ClamAV 0.103.10/27413/Mon Sep 30 10:48:24 2024)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <Zvs9IZ5zxaosKcln@anomaly>
X-Mailman-Original-References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
<CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
<CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
<CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
<4XHQPG4LzsznVwM@mail.python.org>
<Zvrt0RJe5omaFkQq@anomaly>
<CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
<ZvsjI-E8Qtz4rCeL@anomaly>
<4XHbxS5jl4znVGD@mail.python.org>
<CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
View all headers

On 2024-10-01 at 09:09:07 +1000,
Chris Angelico via Python-list <python-list@python.org> wrote:

> On Tue, 1 Oct 2024 at 08:56, Grant Edwards via Python-list
> <python-list@python.org> wrote:
> >
> > On 2024-09-30, Dan Sommers via Python-list <python-list@python.org> wrote:
> >
> > > In Common Lisp, integers can be written in any integer base from two
> > > to thirty six, inclusive. So knowing the last digit doesn't tell
> > > you whether an integer is even or odd until you know the base
> > > anyway.
> >
> > I had to think about that for an embarassingly long time before it
> > clicked.
>
> The only part I'm not clear on is what identifies the base. If you're
> going to write numbers little-endian, it's not that hard to also write
> them with a base indicator before the digits [...]

In Common Lisp, you can write integers as #nnR[digits], where nn is the
decimal representation of the base (possibly without a leading zero),
the # and the R are literal characters, and the digits are written in
the intended base. So the input #16fFFFF is read as the integer 65535.

You can also set or bind the global variable *read-base* (yes, the
asterisks are part of the name) to an integer between 2 and 36, and then
anything that looks like an integer in that base is interpreted as such
(including literals in programs). The literals I described above are
still handled correctly no matter the current value of *read-base*. So
if the value of *read-base* is 16, then the input FFFF is read as the
integer 65535 (as is the input #16rFFFF).

(Pedants may point our details I omitted. I admit to omitting them.)

IIRC, certain [old 8080 and Z-80?] assemblers used to put the base
indicator at the end. So 10 meant, well, 10, but 10H meant 16 and 10b
meant 2 (IDK; the capital H and the lower case b both look right to me).

I don't recall numbers written from least significant digit to most
significant digit (big and little endian *storage*, yes, but not the
digits when presented to or read from a human).

Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
From: Keith Thompson
Newsgroups: comp.lang.python
Organization: None to speak of
Date: Tue, 1 Oct 2024 01:48 UTC
References: 1 2 3 4 5 6 7 8 9 10 11 12
Path: eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: Keith.S.Thompson+u@gmail.com (Keith Thompson)
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
Date: Mon, 30 Sep 2024 18:48:02 -0700
Organization: None to speak of
Lines: 12
Message-ID: <87jzesr3u5.fsf@nosuchdomain.example.com>
References: <CA+hg4RiGjXw3am1s=zVLDpcA-VGS+cWNp_YEyzvS+j2MyDE2Cg@mail.gmail.com>
<CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
<CA+hg4Rhn8iX7rp0uC=MbOi+8g73wQ4y4=uV0dU0jHdDUz3jk4w@mail.gmail.com>
<CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
<4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
<CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
<ZvsjI-E8Qtz4rCeL@anomaly> <4XHbxS5jl4znVGD@mail.python.org>
<CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
<Zvs9IZ5zxaosKcln@anomaly>
<mailman.18.1727741223.3018.python-list@python.org>
MIME-Version: 1.0
Content-Type: text/plain
Injection-Date: Tue, 01 Oct 2024 03:48:02 +0200 (CEST)
Injection-Info: dont-email.me; posting-host="6e12befb7a922ca296630ca4f5b6e746";
logging-data="2708356"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1//WhPlZccEhVp3s4eVShii"
User-Agent: Gnus/5.13 (Gnus v5.13)
Cancel-Lock: sha1:ClATxEYHcpBgQOCWchzApL4KvEY=
sha1:ZMGjz2VAGrLfAUph1XdKMISskqM=
View all headers

2QdxY4RzWzUUiLuE@potatochowder.com writes:
[...]
> In Common Lisp, you can write integers as #nnR[digits], where nn is the
> decimal representation of the base (possibly without a leading zero),
> the # and the R are literal characters, and the digits are written in
> the intended base. So the input #16fFFFF is read as the integer 65535.

Typo: You meant #16RFFFF, not #16fFFFF.

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
void Void(void) { Void(); } /* The recursive call of the void */

Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
From: 2QdxY4RzWzUUiLuE@potatochowder.com
Newsgroups: comp.lang.python
Date: Tue, 1 Oct 2024 15:34 UTC
References: 1 2 3 4 5 6 7 8 9 10 11
Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!fu-berlin.de!uni-berlin.de!not-for-mail
From: 2QdxY4RzWzUUiLuE@potatochowder.com
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
GB) from Kenna API
Date: Tue, 1 Oct 2024 11:34:45 -0400
Lines: 13
Message-ID: <mailman.20.1727796890.3018.python-list@python.org>
References: <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
<4XHQPG4LzsznVwM@mail.python.org> <Zvrt0RJe5omaFkQq@anomaly>
<CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
<ZvsjI-E8Qtz4rCeL@anomaly> <4XHbxS5jl4znVGD@mail.python.org>
<CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
<Zvs9IZ5zxaosKcln@anomaly>
<mailman.18.1727741223.3018.python-list@python.org>
<87jzesr3u5.fsf@nosuchdomain.example.com>
<ZvwWlY-CQg4rUUoO@anomaly>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: news.uni-berlin.de +tBlgfHjrzFhmeBiAvBCIwLF8xppgeFzTEK+kGHwvobQ==
Cancel-Lock: sha1:Vdgz00CYoosicIi6M2wFoyqcchg= sha256:gxQBj38mV0kIF0AdFIkEVBmLQJIrVLxHkd43lDkII6o=
Return-Path: <2QdxY4RzWzUUiLuE@potatochowder.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=potatochowder.com header.i=@potatochowder.com
header.b=G7S0VlHL; dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:API': 0.07;
'-0700,': 0.09; 'characters,': 0.09; 'meant': 0.09; 'received:78':
0.09; 'writes:': 0.09; 'base.': 0.16; 'decimal': 0.16;
'from:addr:2qdxy4rzwzuuilue': 0.16; 'from:addr:potatochowder.com':
0.16; 'integer': 0.16; 'received:136.243': 0.16;
'received:172.58': 0.16; 'received:78.46': 0.16; 'received:www458
.your-server.de': 0.16; 'received:your-server.de': 0.16; 'sorry.':
0.16; 'wrote:': 0.16; 'subject:Help': 0.17; 'to:addr:python-list':
0.20; 'input': 0.21; 'written': 0.22; 'received:de': 0.23;
'python-list': 0.32; 'received:136': 0.32; 'subject:for': 0.33;
'header:In-Reply-To:1': 0.34; 'subject:from': 0.37; 'read': 0.38;
'base': 0.40; 'leading': 0.63; 'skip:# 10': 0.68; 'subject:Data':
0.71; 'subject: \n ': 0.84
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=potatochowder.com; s=default2305; h=In-Reply-To:Content-Type:MIME-Version:
References:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc:
Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:
Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID;
bh=/Trbcu7gptMOJWG5ldKhTNiTC0qLpFRjw47LhSpUFyM=; b=G7S0VlHLhsa365roJ295Jc2Ae4
eCXoUCZWtTIFAW48V+YSpt1kW+qQccTa4w8Imk+Qqp9+9DOFFkxOXCev6CquvjJuCOWZ6TWezyIKL
Xk7DjSweWjl0lOZOzxytJ9+Ot95nvM1vYGBHNx/uU4vmykpeZw0KUy3pABNiJCesEmf/0P1yne/RF
wgkk1IGQiibEH1aVtqNDX5cHD5tFLzDbeMdJiUMMMYOWqLF/Oy3bR/pvX+gqqheRNnUErV9kQqFJB
5mXhgNZrTij4/6y9+J3QiGmf3hWpW6sXPjnOxjPie8S/BzoZRzzWWkbB1vwDSR7WILLdUZZVO2c50
v4b6HZtA==;
Mail-Followup-To: python-list@python.org
Content-Disposition: inline
In-Reply-To: <87jzesr3u5.fsf@nosuchdomain.example.com>
X-Authenticated-Sender: 2QdxY4RzWzUUiLuE@potatochowder.com
X-Virus-Scanned: Clear (ClamAV 0.103.10/27414/Tue Oct 1 10:44:50 2024)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <ZvwWlY-CQg4rUUoO@anomaly>
X-Mailman-Original-References: <CAJQBtgk122sHzs+=MumYM1HW2DwKm1+i02bqgBKh4oUJYievCg@mail.gmail.com>
<4XHQPG4LzsznVwM@mail.python.org>
<Zvrt0RJe5omaFkQq@anomaly>
<CAPTjJmqCz0UthKfs2-sd6E0Jcq23m0r-DgriywwazhWZ381wwg@mail.gmail.com>
<ZvsjI-E8Qtz4rCeL@anomaly>
<4XHbxS5jl4znVGD@mail.python.org>
<CAPTjJmr61ZfAHEUh6NF=EnyLHOTvm_qFrCb75n42Cq_cGdaTWQ@mail.gmail.com>
<Zvs9IZ5zxaosKcln@anomaly>
<mailman.18.1727741223.3018.python-list@python.org>
<87jzesr3u5.fsf@nosuchdomain.example.com>
View all headers

On 2024-09-30 at 18:48:02 -0700,
Keith Thompson via Python-list <python-list@python.org> wrote:

> 2QdxY4RzWzUUiLuE@potatochowder.com writes:
> [...]
> > In Common Lisp, you can write integers as #nnR[digits], where nn is the
> > decimal representation of the base (possibly without a leading zero),
> > the # and the R are literal characters, and the digits are written in
> > the intended base. So the input #16fFFFF is read as the integer 65535.
>
> Typo: You meant #16RFFFF, not #16fFFFF.

Yep. Sorry.

1

rocksolid light 0.9.8
clearnet tor