HomeElementl

up pickle protocol

Description

up pickle protocol

Summary:
User was getting "OSError: [Errno 22] Invalid argument" when the default intermediate storage called pickle.dump with PICKLE_PROTOCOL = 2 which can happen for very large objects. protocol 4 adds support for pickling very large objects: https://docs.python.org/3/library/pickle.html#pickle-protocols

there are potential issues if the pickle writer gets ahead of the pickle reader. I think there are some rare edge cases where this is unsafe, such as if you use python2 to re-execute a pipeline run that was originally executed in python3

separately "OSError: [Errno 22] Invalid argument" from the default intermediate storage is a pretty confusing error and it's not clear for the user what they should do next (set up a diff intermediate storage)

Test Plan: bk

Reviewers: alangenfeld, max, sandyryza, yuhan

Reviewed By: yuhan

Differential Revision: https://dagster.phacility.com/D4937

Details

Provenance
catherinewuAuthored on Oct 28 2020, 3:54 PM
Reviewer
yuhan
Differential Revision
D4937: up pickle protocol
Parents
R1:b67ab070dcfb: (refactor-input-source-to-thunk-4) Add checks to catch errors when new sourceā€¦
Branches
Unknown
Tags
Unknown