How to use System.console as Apache Flink source
Apache Flink provides a lot of different streaming sources but I was wondering whether it's possible to use the Console as a source of data. I haven't found any examples online.
I've come up with this:
DataStream<String> consoleInput = flinkEnv.addSource(new SourceFunction<String>() {
@Override
public void run(SourceContext<String> ctx) throws Exception {
Scanner sc = new Scanner(System.in);
while (true)
ctx.collect(sc.nextLine());
}
@Override
public void cancel() {
}
});
I was wondering if this is ok or is there a better way.
java apache-flink flink-streaming
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
add a comment |
Apache Flink provides a lot of different streaming sources but I was wondering whether it's possible to use the Console as a source of data. I haven't found any examples online.
I've come up with this:
DataStream<String> consoleInput = flinkEnv.addSource(new SourceFunction<String>() {
@Override
public void run(SourceContext<String> ctx) throws Exception {
Scanner sc = new Scanner(System.in);
while (true)
ctx.collect(sc.nextLine());
}
@Override
public void cancel() {
}
});
I was wondering if this is ok or is there a better way.
java apache-flink flink-streaming
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
add a comment |
Apache Flink provides a lot of different streaming sources but I was wondering whether it's possible to use the Console as a source of data. I haven't found any examples online.
I've come up with this:
DataStream<String> consoleInput = flinkEnv.addSource(new SourceFunction<String>() {
@Override
public void run(SourceContext<String> ctx) throws Exception {
Scanner sc = new Scanner(System.in);
while (true)
ctx.collect(sc.nextLine());
}
@Override
public void cancel() {
}
});
I was wondering if this is ok or is there a better way.
java apache-flink flink-streaming
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
Apache Flink provides a lot of different streaming sources but I was wondering whether it's possible to use the Console as a source of data. I haven't found any examples online.
I've come up with this:
DataStream<String> consoleInput = flinkEnv.addSource(new SourceFunction<String>() {
@Override
public void run(SourceContext<String> ctx) throws Exception {
Scanner sc = new Scanner(System.in);
while (true)
ctx.collect(sc.nextLine());
}
@Override
public void cancel() {
}
});
I was wondering if this is ok or is there a better way.
java apache-flink flink-streaming
java apache-flink flink-streaming
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
edited Jan 18 at 17:41
AsadSMalik
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
asked Jan 18 at 17:20
AsadSMalikAsadSMalik
318825
318825
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Using the console as a data source is certainly doable, but has the issue that your application won't be fault tolerant, since Flink won't be able rewind and replay the input stream in the event of a failure.
But this often done for prototypes and experiments, typically with a SocketTextStreamFunction, as in
env.addSource(new SocketTextStreamFunction("localhost", 9999, "n", -1))
You can then use netcat to attach the console to port 9999
nc -lk 9999
or, as some versions of netcat require,
nc -l -p 9000
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54258702%2fhow-to-use-system-console-as-apache-flink-source%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Using the console as a data source is certainly doable, but has the issue that your application won't be fault tolerant, since Flink won't be able rewind and replay the input stream in the event of a failure.
But this often done for prototypes and experiments, typically with a SocketTextStreamFunction, as in
env.addSource(new SocketTextStreamFunction("localhost", 9999, "n", -1))
You can then use netcat to attach the console to port 9999
nc -lk 9999
or, as some versions of netcat require,
nc -l -p 9000
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
add a comment |
Using the console as a data source is certainly doable, but has the issue that your application won't be fault tolerant, since Flink won't be able rewind and replay the input stream in the event of a failure.
But this often done for prototypes and experiments, typically with a SocketTextStreamFunction, as in
env.addSource(new SocketTextStreamFunction("localhost", 9999, "n", -1))
You can then use netcat to attach the console to port 9999
nc -lk 9999
or, as some versions of netcat require,
nc -l -p 9000
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
add a comment |
Using the console as a data source is certainly doable, but has the issue that your application won't be fault tolerant, since Flink won't be able rewind and replay the input stream in the event of a failure.
But this often done for prototypes and experiments, typically with a SocketTextStreamFunction, as in
env.addSource(new SocketTextStreamFunction("localhost", 9999, "n", -1))
You can then use netcat to attach the console to port 9999
nc -lk 9999
or, as some versions of netcat require,
nc -l -p 9000
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
Using the console as a data source is certainly doable, but has the issue that your application won't be fault tolerant, since Flink won't be able rewind and replay the input stream in the event of a failure.
But this often done for prototypes and experiments, typically with a SocketTextStreamFunction, as in
env.addSource(new SocketTextStreamFunction("localhost", 9999, "n", -1))
You can then use netcat to attach the console to port 9999
nc -lk 9999
or, as some versions of netcat require,
nc -l -p 9000
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
answered Jan 18 at 18:17
David AndersonDavid Anderson
5,37421321
5,37421321
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
add a comment |
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
Yes I saw this in the tutorials. I wanted to use the console for experimenting :)
– AsadSMalik
Jan 18 at 20:22
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54258702%2fhow-to-use-system-console-as-apache-flink-source%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
