Python 3.6: Why does pickle.dumps(nparray) permanently increase refcount?
An np.ndarray, when pickled, increments reference counter from the dumps function, however the ref count is never decremented.
Python 3.6.4 Anaconda
Ubuntu 16.04.5 LTS
numpy 1.16.0
I have already tried converting to a list using numpy.array.tolist() however this method is far too slow.
import numpy as np
import pickle
import sys
a = np.ndarray((10, 10), dtype=np.uint8)
print(sys.getrefcount(a)) # 2
pickle.dumps(a)
print(sys.getrefcount(a)) # 3
I would expect the output to be 2, 2 due to the Py_DECREF that occurs in the pickler dumps function, however it remains.
Output is 2, 3 and I cannot fix it. I am leaking memory like crazy.
Currently digging into _pickle.c.
python numpy automatic-ref-counting
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
|
show 2 more comments
An np.ndarray, when pickled, increments reference counter from the dumps function, however the ref count is never decremented.
Python 3.6.4 Anaconda
Ubuntu 16.04.5 LTS
numpy 1.16.0
I have already tried converting to a list using numpy.array.tolist() however this method is far too slow.
import numpy as np
import pickle
import sys
a = np.ndarray((10, 10), dtype=np.uint8)
print(sys.getrefcount(a)) # 2
pickle.dumps(a)
print(sys.getrefcount(a)) # 3
I would expect the output to be 2, 2 due to the Py_DECREF that occurs in the pickler dumps function, however it remains.
Output is 2, 3 and I cannot fix it. I am leaking memory like crazy.
Currently digging into _pickle.c.
python numpy automatic-ref-counting
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
frameis just rather bog-standardnumpy.ndarray()instance. Can you reproduce this issue withoutcv2and just with, say,frame = np.zeros((10, 10), dtype=np.uint8)?
– Martijn Pieters♦
Jan 19 at 16:33
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59
|
show 2 more comments
An np.ndarray, when pickled, increments reference counter from the dumps function, however the ref count is never decremented.
Python 3.6.4 Anaconda
Ubuntu 16.04.5 LTS
numpy 1.16.0
I have already tried converting to a list using numpy.array.tolist() however this method is far too slow.
import numpy as np
import pickle
import sys
a = np.ndarray((10, 10), dtype=np.uint8)
print(sys.getrefcount(a)) # 2
pickle.dumps(a)
print(sys.getrefcount(a)) # 3
I would expect the output to be 2, 2 due to the Py_DECREF that occurs in the pickler dumps function, however it remains.
Output is 2, 3 and I cannot fix it. I am leaking memory like crazy.
Currently digging into _pickle.c.
python numpy automatic-ref-counting
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
An np.ndarray, when pickled, increments reference counter from the dumps function, however the ref count is never decremented.
Python 3.6.4 Anaconda
Ubuntu 16.04.5 LTS
numpy 1.16.0
I have already tried converting to a list using numpy.array.tolist() however this method is far too slow.
import numpy as np
import pickle
import sys
a = np.ndarray((10, 10), dtype=np.uint8)
print(sys.getrefcount(a)) # 2
pickle.dumps(a)
print(sys.getrefcount(a)) # 3
I would expect the output to be 2, 2 due to the Py_DECREF that occurs in the pickler dumps function, however it remains.
Output is 2, 3 and I cannot fix it. I am leaking memory like crazy.
Currently digging into _pickle.c.
python numpy automatic-ref-counting
python numpy automatic-ref-counting
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
edited Jan 19 at 17:06
Eddie Callahan
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
asked Jan 19 at 16:07
Eddie CallahanEddie Callahan
104
104
frameis just rather bog-standardnumpy.ndarray()instance. Can you reproduce this issue withoutcv2and just with, say,frame = np.zeros((10, 10), dtype=np.uint8)?
– Martijn Pieters♦
Jan 19 at 16:33
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59
|
show 2 more comments
frameis just rather bog-standardnumpy.ndarray()instance. Can you reproduce this issue withoutcv2and just with, say,frame = np.zeros((10, 10), dtype=np.uint8)?
– Martijn Pieters♦
Jan 19 at 16:33
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59
frame is just rather bog-standard numpy.ndarray() instance. Can you reproduce this issue without cv2 and just with, say, frame = np.zeros((10, 10), dtype=np.uint8)?– Martijn Pieters♦
Jan 19 at 16:33
frame is just rather bog-standard numpy.ndarray() instance. Can you reproduce this issue without cv2 and just with, say, frame = np.zeros((10, 10), dtype=np.uint8)?– Martijn Pieters♦
Jan 19 at 16:33
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59
|
show 2 more comments
1 Answer
1
active
oldest
votes
You've run into this specific bug, and it's a regression in Numpy 1.16.0 only. New code to add support for a new pickle protocol 5 was leaking references to a bound __reduce__ method in the fallback case.
You can either wait for that bug to be fixed and 1.16.1 to be released, or go back to Numpy 1.15.4.
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54268926%2fpython-3-6-why-does-pickle-dumpsnparray-permanently-increase-refcount%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You've run into this specific bug, and it's a regression in Numpy 1.16.0 only. New code to add support for a new pickle protocol 5 was leaking references to a bound __reduce__ method in the fallback case.
You can either wait for that bug to be fixed and 1.16.1 to be released, or go back to Numpy 1.15.4.
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
add a comment |
You've run into this specific bug, and it's a regression in Numpy 1.16.0 only. New code to add support for a new pickle protocol 5 was leaking references to a bound __reduce__ method in the fallback case.
You can either wait for that bug to be fixed and 1.16.1 to be released, or go back to Numpy 1.15.4.
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
add a comment |
You've run into this specific bug, and it's a regression in Numpy 1.16.0 only. New code to add support for a new pickle protocol 5 was leaking references to a bound __reduce__ method in the fallback case.
You can either wait for that bug to be fixed and 1.16.1 to be released, or go back to Numpy 1.15.4.
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
You've run into this specific bug, and it's a regression in Numpy 1.16.0 only. New code to add support for a new pickle protocol 5 was leaking references to a bound __reduce__ method in the fallback case.
You can either wait for that bug to be fixed and 1.16.1 to be released, or go back to Numpy 1.15.4.
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
edited Jan 19 at 23:18
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
answered Jan 19 at 17:08
Martijn Pieters♦Martijn Pieters
709k13624752297
709k13624752297
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
add a comment |
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
Thanks so much!
– Eddie Callahan
Jan 19 at 17:11
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54268926%2fpython-3-6-why-does-pickle-dumpsnparray-permanently-increase-refcount%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
frameis just rather bog-standardnumpy.ndarray()instance. Can you reproduce this issue withoutcv2and just with, say,frame = np.zeros((10, 10), dtype=np.uint8)?– Martijn Pieters♦
Jan 19 at 16:33
Also, what exact version of Python 3.6 is this, and what OS is this on?
– Martijn Pieters♦
Jan 19 at 16:35
@MartijnPieters I am on Python 3.6.8 Anaconda ubuntu 16.04.5 LTS. I will try to reproduce with np.zeros. Thanks for the advice! Will comment again with results.
– Eddie Callahan
Jan 19 at 16:52
@MartijnPieters I was able to reproduce with just np.zeros. Numpy refcount increments without ever decrementing. This even ocurrs with ndarray.dumps()
– Eddie Callahan
Jan 19 at 16:56
right, so we are getting somewhere a little simpler. Can you update your question? You may want to consider filing a bug report with the numpy project, however.
– Martijn Pieters♦
Jan 19 at 16:59