Is it possible to limit iterations (b.N) in Go Benchmarking?
I'm trying to benchmark an operation which requires expensive preparation which I'm excluding using StopTimer() and StartTimer(). Specifically, I'm benchmarking nth-item insertion into a sorted list.
Sample code:
n := 100
// Run the process b.N times
for i := 0; i < b.N; i++ {
// Stop the timer for our expensive preparation work (inserting n-1 items)
b.StopTimer()
// ...
// Insert n-1 items
for j := 1; j < n; j++ {
m.InsertItem(o)
}
// ...
// Resume the timer
b.StartTimer()
// Insert the nth item
m.InsertItem(o)
}
The problem is that Go's benchmarking heuristic is limiting b.N according to the benchmarked time, not the total time. It ends up asking for 5MM (5000000) iterations of 100th order insertion, and this takes more time than is reasonable (I want to benchmark up to the 10 millionth item insertion).
Is there a way to specify a maximum b.N for a specific benchmark in Go's benchmark tool? I did not find anything in the docs myself.
go
asked Nov 15 '18 at 15:39
gweltergwelter
11017
add a comment |
I'm trying to benchmark an operation which requires expensive preparation which I'm excluding using StopTimer() and StartTimer(). Specifically, I'm benchmarking nth-item insertion into a sorted list.
Sample code:
n := 100
// Run the process b.N times
for i := 0; i < b.N; i++ {
// Stop the timer for our expensive preparation work (inserting n-1 items)
b.StopTimer()
// ...
// Insert n-1 items
for j := 1; j < n; j++ {
m.InsertItem(o)
}
// ...
// Resume the timer
b.StartTimer()
// Insert the nth item
m.InsertItem(o)
}
The problem is that Go's benchmarking heuristic is limiting b.N according to the benchmarked time, not the total time. It ends up asking for 5MM (5000000) iterations of 100th order insertion, and this takes more time than is reasonable (I want to benchmark up to the 10 millionth item insertion).
Is there a way to specify a maximum b.N for a specific benchmark in Go's benchmark tool? I did not find anything in the docs myself.
go
asked Nov 15 '18 at 15:39
gweltergwelter
11017
add a comment |
I'm trying to benchmark an operation which requires expensive preparation which I'm excluding using StopTimer() and StartTimer(). Specifically, I'm benchmarking nth-item insertion into a sorted list.
Sample code:
n := 100
// Run the process b.N times
for i := 0; i < b.N; i++ {
// Stop the timer for our expensive preparation work (inserting n-1 items)
b.StopTimer()
// ...
// Insert n-1 items
for j := 1; j < n; j++ {
m.InsertItem(o)
}
// ...
// Resume the timer
b.StartTimer()
// Insert the nth item
m.InsertItem(o)
}
The problem is that Go's benchmarking heuristic is limiting b.N according to the benchmarked time, not the total time. It ends up asking for 5MM (5000000) iterations of 100th order insertion, and this takes more time than is reasonable (I want to benchmark up to the 10 millionth item insertion).
Is there a way to specify a maximum b.N for a specific benchmark in Go's benchmark tool? I did not find anything in the docs myself.
go
asked Nov 15 '18 at 15:39
gweltergwelter
11017
I'm trying to benchmark an operation which requires expensive preparation which I'm excluding using StopTimer() and StartTimer(). Specifically, I'm benchmarking nth-item insertion into a sorted list.
Sample code:
n := 100
// Run the process b.N times
for i := 0; i < b.N; i++ {
// Stop the timer for our expensive preparation work (inserting n-1 items)
b.StopTimer()
// ...
// Insert n-1 items
for j := 1; j < n; j++ {
m.InsertItem(o)
}
// ...
// Resume the timer
b.StartTimer()
// Insert the nth item
m.InsertItem(o)
}
The problem is that Go's benchmarking heuristic is limiting b.N according to the benchmarked time, not the total time. It ends up asking for 5MM (5000000) iterations of 100th order insertion, and this takes more time than is reasonable (I want to benchmark up to the 10 millionth item insertion).
Is there a way to specify a maximum b.N for a specific benchmark in Go's benchmark tool? I did not find anything in the docs myself.
go
go
asked Nov 15 '18 at 15:39
gweltergwelter
11017
asked Nov 15 '18 at 15:39
gweltergwelter
11017
edited Nov 15 '18 at 16:02
gwelter
asked Nov 15 '18 at 15:39
gweltergwelter
11017
asked Nov 15 '18 at 15:39
gweltergwelter
11017
asked Nov 15 '18 at 15:39
gweltergwelter
11017
11017
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
-benchtime flag can be used to specify the time for which you want to run the benchmark(source). Although I haven't tried it, providing a lower value of time than 1 sec should do the trick.
go test -bench=. -benchtime=10s
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53322925%2fis-it-possible-to-limit-iterations-b-n-in-go-benchmarking%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
-benchtime flag can be used to specify the time for which you want to run the benchmark(source). Although I haven't tried it, providing a lower value of time than 1 sec should do the trick.
go test -bench=. -benchtime=10s
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
add a comment |
-benchtime flag can be used to specify the time for which you want to run the benchmark(source). Although I haven't tried it, providing a lower value of time than 1 sec should do the trick.
go test -bench=. -benchtime=10s
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
add a comment |
-benchtime flag can be used to specify the time for which you want to run the benchmark(source). Although I haven't tried it, providing a lower value of time than 1 sec should do the trick.
go test -bench=. -benchtime=10s
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
-benchtime flag can be used to specify the time for which you want to run the benchmark(source). Although I haven't tried it, providing a lower value of time than 1 sec should do the trick.
go test -bench=. -benchtime=10s
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
answered Nov 15 '18 at 16:22
Saty AnandSaty Anand
409211
409211
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
add a comment |
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
I was hopeful about this one, but it looks like that's actually a minimum time to hit (according to the [docs re: benchmark flag: "Run enough iterations of each benchmark to take t". Even worse for my case, I tried it out and it looks like it's trying to hit that minimum run time based on measured benchmark time, not total execution time. So it's taking even longer. Thanks for the suggestion, though, it was the right idea! golang.org/cmd/go/#hdr-Testing_flags
– gwelter
Nov 15 '18 at 16:55
1
1
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
Actually, I tested this further and it appears that -benchtime does behave as a target, not just a min or max. Since the per op measured execution time for my case is around 300ns, I set benchtime to 1ms and it worked reasonably well. It's a blunt instrument since you can't set benchtime per benchmark, but it gets the job done. Thank you Saty!
– gwelter
Nov 15 '18 at 17:44
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53322925%2fis-it-possible-to-limit-iterations-b-n-in-go-benchmarking%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
