talk-data.com talk-data.com

PyData talk 2025-12-10 at 19:15

Processing large JSON files without running out of memory

Description

If you need to process a large JSON file in Python, it’s very easy to run out of memory while loading the data, leading to a super-slow run time or out-of-memory crashes. In this talk you'll learn:

  • How to measure memory usage.
  • Why loading JSON takes a lot of memory.
  • Four different ways to reduce memory usage when loading large JSON files.