Rocksolid Light

News from da outaworlds

mail  files  register  groups  login

Message-ID:  

Q: What happens when four WASPs find themselves in the same room? A: A dinner party.


comp / comp.lang.python / Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

SubjectAuthor
o Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from KeBarry

1
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
From: Barry
Newsgroups: comp.lang.python
Date: Mon, 30 Sep 2024 15:30 UTC
References: 1 2
Path: eternal-september.org!news.eternal-september.org!feeder3.eternal-september.org!fu-berlin.de!uni-berlin.de!not-for-mail
From: barry@barrys-emacs.org (Barry)
Newsgroups: comp.lang.python
Subject: Re: Help with Streaming and Chunk Processing for Large JSON Data (60
GB) from Kenna API
Date: Mon, 30 Sep 2024 16:30:19 +0100
Lines: 19
Message-ID: <mailman.5.1727710877.3018.python-list@python.org>
References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
<082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org>
Mime-Version: 1.0 (1.0)
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de 6e+Y8nVHxPxvtyTP1BC3MA3IQzAXSYvI2GI9lMTHE91Q==
Cancel-Lock: sha1:R944ysFvUuhbzx/bgIIgFM7S0hU= sha256:AZ2vcwBbz67OGBgqZefOct08IL35PZ4nNUgPR6QbiZM=
Return-Path: <barry@barrys-emacs.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'subject:API': 0.07;
'cc:addr:python-list': 0.09; 'from:addr:barry': 0.09;
'received:217.70': 0.09; 'received:217.70.178': 0.09;
'received:gandi.net': 0.09; 'received:mail.gandi.net': 0.09;
'cc:no real name:2**0': 0.14; 'import': 0.15; 'barry': 0.16;
'from:addr:barrys-emacs.org': 0.16; 'janhangeer': 0.16; 'message-
id:@barrys-emacs.org': 0.16; 'wrote:': 0.16; 'subject:Help': 0.17;
'cc:addr:python.org': 0.20; 'cc:2**1': 0.23; 'computer': 0.29;
'python-list': 0.32; 'sep': 0.32; 'unless': 0.32; 'subject:for':
0.33; 'header:In-Reply-To:1': 0.34; 'subject:from': 0.37;
'received:217': 0.67; 'subject:Data': 0.71
In-Reply-To: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
X-Mailer: iPad Mail (22A3354)
X-GND-Sasl: barry@barrys-emacs.org
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <082705B5-7C14-4D33-BF38-73F9CB166293@barrys-emacs.org>
X-Mailman-Original-References: <CADrxXXmHUwsQbWqNrwzyKWLyTK0J3Hf0z8hAhGwKYoF2PwK7QA@mail.gmail.com>
View all headers

> On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list@python.org> wrote:
>
>
> import polars as pl
> pl.read_json("file.json")
>
>

This is not going to work unless the computer has a lot more the 60GiB of RAM.

As later suggested a streaming parser is required.

Barry

1

rocksolid light 0.9.8
clearnet tor