spark repartition to one output file per customer
1
Assume I have a dataframe like:
client_id,report_date,date,value_1,value_2
1,2019-01-01,2019-01-01,1,2
1,2019-01-01,2019-01-02,3,4
1,2019-01-01,2019-01-03,5,6
2,2019-01-01,2019-01-01,1,2
2,2019-01-01,2019-01-02,3,4
2,2019-01-01,2019-01-03,5,6
My desired output structure would be a CSV or JSON with:
results/
client_id=1/
report_date=2019-01-01
<<somename>>.csv
client_id=2/
report_date=2019-01-01
<<somename>>.csv
To achieve this I use
df.repartition(2, "customer_id", "report_date")
.sortWithinPartitions("date", "value1")
.write.partitionBy("customer_id", "report_date")
.csv(...)
However, instead of the desired single file per client and report date (partition) I end up with two.
Spark SQL - Difference between df.repartition and DataFrameWriter partitionBy? explains why.
However, using a repartition(1) would work. But in case the number of customer_id is large could run into OOM. Is there still a way to achieve the desired result? The file per client_id is small.
data-partitioning
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
|
show 20 more comments
1
Assume I have a dataframe like:
client_id,report_date,date,value_1,value_2
1,2019-01-01,2019-01-01,1,2
1,2019-01-01,2019-01-02,3,4
1,2019-01-01,2019-01-03,5,6
2,2019-01-01,2019-01-01,1,2
2,2019-01-01,2019-01-02,3,4
2,2019-01-01,2019-01-03,5,6
My desired output structure would be a CSV or JSON with:
results/
client_id=1/
report_date=2019-01-01
<<somename>>.csv
client_id=2/
report_date=2019-01-01
<<somename>>.csv
To achieve this I use
df.repartition(2, "customer_id", "report_date")
.sortWithinPartitions("date", "value1")
.write.partitionBy("customer_id", "report_date")
.csv(...)
However, instead of the desired single file per client and report date (partition) I end up with two.
Spark SQL - Difference between df.repartition and DataFrameWriter partitionBy? explains why.
However, using a repartition(1) would work. But in case the number of customer_id is large could run into OOM. Is there still a way to achieve the desired result? The file per client_id is small.
data-partitioning
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
That is correct. But differentiatereport_dateanddate. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.
– Georg Heiler
Jan 20 at 18:24
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57
|
show 20 more comments
1
1
1
Assume I have a dataframe like:
client_id,report_date,date,value_1,value_2
1,2019-01-01,2019-01-01,1,2
1,2019-01-01,2019-01-02,3,4
1,2019-01-01,2019-01-03,5,6
2,2019-01-01,2019-01-01,1,2
2,2019-01-01,2019-01-02,3,4
2,2019-01-01,2019-01-03,5,6
My desired output structure would be a CSV or JSON with:
results/
client_id=1/
report_date=2019-01-01
<<somename>>.csv
client_id=2/
report_date=2019-01-01
<<somename>>.csv
To achieve this I use
df.repartition(2, "customer_id", "report_date")
.sortWithinPartitions("date", "value1")
.write.partitionBy("customer_id", "report_date")
.csv(...)
However, instead of the desired single file per client and report date (partition) I end up with two.
Spark SQL - Difference between df.repartition and DataFrameWriter partitionBy? explains why.
However, using a repartition(1) would work. But in case the number of customer_id is large could run into OOM. Is there still a way to achieve the desired result? The file per client_id is small.
data-partitioning
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
Assume I have a dataframe like:
client_id,report_date,date,value_1,value_2
1,2019-01-01,2019-01-01,1,2
1,2019-01-01,2019-01-02,3,4
1,2019-01-01,2019-01-03,5,6
2,2019-01-01,2019-01-01,1,2
2,2019-01-01,2019-01-02,3,4
2,2019-01-01,2019-01-03,5,6
My desired output structure would be a CSV or JSON with:
results/
client_id=1/
report_date=2019-01-01
<<somename>>.csv
client_id=2/
report_date=2019-01-01
<<somename>>.csv
To achieve this I use
df.repartition(2, "customer_id", "report_date")
.sortWithinPartitions("date", "value1")
.write.partitionBy("customer_id", "report_date")
.csv(...)
However, instead of the desired single file per client and report date (partition) I end up with two.
Spark SQL - Difference between df.repartition and DataFrameWriter partitionBy? explains why.
However, using a repartition(1) would work. But in case the number of customer_id is large could run into OOM. Is there still a way to achieve the desired result? The file per client_id is small.
data-partitioning
data-partitioning
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
asked Jan 19 at 9:30
Georg HeilerGeorg Heiler
5,085653129
5,085653129
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
That is correct. But differentiatereport_dateanddate. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.
– Georg Heiler
Jan 20 at 18:24
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57
|
show 20 more comments
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
That is correct. But differentiatereport_dateanddate. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.
– Georg Heiler
Jan 20 at 18:24
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
That is correct. But differentiate
report_date and date. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.– Georg Heiler
Jan 20 at 18:24
That is correct. But differentiate
report_date and date. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.– Georg Heiler
Jan 20 at 18:24
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57
|
show 20 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54265725%2fspark-repartition-to-one-output-file-per-customer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54265725%2fspark-repartition-to-one-output-file-per-customer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
So, what is the question? Customer_id is large but then you state is small...
– thebluephantom
Jan 20 at 17:40
The data per customer is mall but there are many customers. I want to end up with one single file per customer. My initial strategy was to use spark partitions. However they would only work if repartition 1 is executed. But this will not work as there are too many customers. So is there a different option?
– Georg Heiler
Jan 20 at 18:11
But in your example that is what I see. What if there are 2 dates for the same customer?
– thebluephantom
Jan 20 at 18:14
That is correct. But differentiate
report_dateanddate. Partitioning os happending per customer_id and 'report_date' i.e. there should be a file per customer_id and report_date.– Georg Heiler
Jan 20 at 18:24
But that's what I see happening, I just tried it and got that as you state. So, what to think? On a small sample.
– thebluephantom
Jan 20 at 18:57