Real time report generate issue
We are developing a application has 2.2 million customers. We develop rest api. Transaction report api hit huge table to generate a complex report. If a single time more than 100k customers get this data huge hit my database server. My question is which way we generate report without hit SQL server database. Or world best organization how to manage this type of application database.
We use asp.net mvc5 and SQL server
sql-server elasticsearch edited Feb 8 at 16:56
Cœur
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
add a comment |
We are developing a application has 2.2 million customers. We develop rest api. Transaction report api hit huge table to generate a complex report. If a single time more than 100k customers get this data huge hit my database server. My question is which way we generate report without hit SQL server database. Or world best organization how to manage this type of application database.
We use asp.net mvc5 and SQL server
sql-server elasticsearch edited Feb 8 at 16:56
Cœur
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45
add a comment |
We are developing a application has 2.2 million customers. We develop rest api. Transaction report api hit huge table to generate a complex report. If a single time more than 100k customers get this data huge hit my database server. My question is which way we generate report without hit SQL server database. Or world best organization how to manage this type of application database.
We use asp.net mvc5 and SQL server
sql-server elasticsearch edited Feb 8 at 16:56
Cœur
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
We are developing a application has 2.2 million customers. We develop rest api. Transaction report api hit huge table to generate a complex report. If a single time more than 100k customers get this data huge hit my database server. My question is which way we generate report without hit SQL server database. Or world best organization how to manage this type of application database.
We use asp.net mvc5 and SQL server
sql-server elasticsearch sql-server elasticsearch edited Feb 8 at 16:56
Cœur
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
edited Feb 8 at 16:56
Cœur
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
edited Feb 8 at 16:56
Cœur
18.5k9110148
edited Feb 8 at 16:56
Cœur
18.5k9110148
edited Feb 8 at 16:56
Cœur
18.5k9110148
18.5k9110148
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
asked Nov 15 '18 at 8:28
tapos ghoshtapos ghosh
1,4571121
1,4571121
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45
add a comment |
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45
add a comment |
2 Answers
2
active
oldest
votes
There are at least two standard methods to "generate report without hit SQL server database".
- use snapshot isolation in the same OLTP database to avoid the report locking other writing transactions and ensure to get consistent data
- use the data transferred (ETL, log shipping, replication...) from OLTP to other (OLAP) database (i.e. to a warehouse or later in datamarts/cubes)
answered Nov 15 '18 at 9:26
sergeserge
69147
add a comment |
you can have install other SQL instance in prod and create transition replication on the database. Report on the replicated side on the database.
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315172%2freal-time-report-generate-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
There are at least two standard methods to "generate report without hit SQL server database".
- use snapshot isolation in the same OLTP database to avoid the report locking other writing transactions and ensure to get consistent data
- use the data transferred (ETL, log shipping, replication...) from OLTP to other (OLAP) database (i.e. to a warehouse or later in datamarts/cubes)
answered Nov 15 '18 at 9:26
sergeserge
69147
add a comment |
There are at least two standard methods to "generate report without hit SQL server database".
- use snapshot isolation in the same OLTP database to avoid the report locking other writing transactions and ensure to get consistent data
- use the data transferred (ETL, log shipping, replication...) from OLTP to other (OLAP) database (i.e. to a warehouse or later in datamarts/cubes)
answered Nov 15 '18 at 9:26
sergeserge
69147
add a comment |
There are at least two standard methods to "generate report without hit SQL server database".
- use snapshot isolation in the same OLTP database to avoid the report locking other writing transactions and ensure to get consistent data
- use the data transferred (ETL, log shipping, replication...) from OLTP to other (OLAP) database (i.e. to a warehouse or later in datamarts/cubes)
answered Nov 15 '18 at 9:26
sergeserge
69147
There are at least two standard methods to "generate report without hit SQL server database".
- use snapshot isolation in the same OLTP database to avoid the report locking other writing transactions and ensure to get consistent data
- use the data transferred (ETL, log shipping, replication...) from OLTP to other (OLAP) database (i.e. to a warehouse or later in datamarts/cubes)
answered Nov 15 '18 at 9:26
sergeserge
69147
answered Nov 15 '18 at 9:26
sergeserge
69147
answered Nov 15 '18 at 9:26
sergeserge
69147
answered Nov 15 '18 at 9:26
sergeserge
69147
69147
add a comment |
add a comment |
you can have install other SQL instance in prod and create transition replication on the database. Report on the replicated side on the database.
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
add a comment |
you can have install other SQL instance in prod and create transition replication on the database. Report on the replicated side on the database.
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
add a comment |
you can have install other SQL instance in prod and create transition replication on the database. Report on the replicated side on the database.
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
you can have install other SQL instance in prod and create transition replication on the database. Report on the replicated side on the database.
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
answered Nov 15 '18 at 14:35
Hiten004Hiten004
2,0411530
2,0411530
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53315172%2freal-time-report-generate-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Have you exhausted all avenues around indexes, partitioning? What kind of latency can you handle (1 second? 1 minute?) Does the report need to cross all of the data?
– Nick.McDermaid
Nov 15 '18 at 9:12
Nick. My thinking is, if I store data 2 separate database one is SQL server other is elasticsearch. Then all report i generate from eleasticSearch . Is it good decision
– tapos ghosh
Nov 15 '18 at 12:24
It's not really a good idea to use Indian words here. People will definitely get confused about what is a "22lac customer".
– James Z
Nov 15 '18 at 14:24
It depends on the type of reports and how much latency you can deal with. Neither of which are described.
– Nick.McDermaid
Nov 17 '18 at 2:04
An elastic search report is very different to a relational report. What is this complex report? Is it an aggregate of figures or is it text?
– Nick.McDermaid
Nov 20 '18 at 9:45