Elasticsearch curator delete action - delete “n” older indices after rollover?

The following elasticsearch curator (below) set-up in curator-actions.yml is configured to delete indices based on age filter but I would like to set-up a config that works with rollover as follows:

Only keep the last index and new created rolled-over index -> thus deleting all other indices after a rollover is successful. What is the best way to accomplish this?? Possible with NEST in code?

Here is my current delete action...any help is much appreciated, thanks!

in curator-actions.yml

action: delete_indices

description: >-

  Delete indices older than 3 days (based on index creation date)

options:

  ignore_empty_list: True

  continue_if_exception: True

filters:

- filtertype: pattern

  kind: prefix

  value: applogging-test

- filtertype: age

  source: creation_date

  direction: older

  unit: days

  unit_count: 3

asked Nov 15 '18 at 13:13

DdRegalo

405

Can you elaborate the rollover part please ? Dont think I get it. Can’t you just schedule curator to run after rollover ?

– ben5556
Nov 15 '18 at 19:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg: PUT /logs-000001 { "aliases": { "logs_write": {} } } # Add > 1000 documents to logs-000001 POST /logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000, "max_size": "5gb" } } Now, using the curator I intended to config a delete action that deletes all indices except the most recent 2 (eg. the newest index created by the rollover and the latest index that was just rolled over). I had a go with a count filter type but not got it yet!

– DdRegalo
Nov 16 '18 at 9:06

add a comment |

Only keep the last index and new created rolled-over index -> thus deleting all other indices after a rollover is successful. What is the best way to accomplish this?? Possible with NEST in code?

Here is my current delete action...any help is much appreciated, thanks!

in curator-actions.yml

action: delete_indices

description: >-

  Delete indices older than 3 days (based on index creation date)

options:

  ignore_empty_list: True

  continue_if_exception: True

filters:

- filtertype: pattern

  kind: prefix

  value: applogging-test

- filtertype: age

  source: creation_date

  direction: older

  unit: days

  unit_count: 3

asked Nov 15 '18 at 13:13

DdRegalo

405

Can you elaborate the rollover part please ? Dont think I get it. Can’t you just schedule curator to run after rollover ?

– ben5556
Nov 15 '18 at 19:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg: PUT /logs-000001 { "aliases": { "logs_write": {} } } # Add > 1000 documents to logs-000001 POST /logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000, "max_size": "5gb" } } Now, using the curator I intended to config a delete action that deletes all indices except the most recent 2 (eg. the newest index created by the rollover and the latest index that was just rolled over). I had a go with a count filter type but not got it yet!

– DdRegalo
Nov 16 '18 at 9:06

add a comment |

Only keep the last index and new created rolled-over index -> thus deleting all other indices after a rollover is successful. What is the best way to accomplish this?? Possible with NEST in code?

Here is my current delete action...any help is much appreciated, thanks!

in curator-actions.yml

action: delete_indices

description: >-

  Delete indices older than 3 days (based on index creation date)

options:

  ignore_empty_list: True

  continue_if_exception: True

filters:

- filtertype: pattern

  kind: prefix

  value: applogging-test

- filtertype: age

  source: creation_date

  direction: older

  unit: days

  unit_count: 3

asked Nov 15 '18 at 13:13

DdRegalo

405

Only keep the last index and new created rolled-over index -> thus deleting all other indices after a rollover is successful. What is the best way to accomplish this?? Possible with NEST in code?

Here is my current delete action...any help is much appreciated, thanks!

in curator-actions.yml

action: delete_indices

description: >-

  Delete indices older than 3 days (based on index creation date)

options:

  ignore_empty_list: True

  continue_if_exception: True

filters:

- filtertype: pattern

  kind: prefix

  value: applogging-test

- filtertype: age

  source: creation_date

  direction: older

  unit: days

  unit_count: 3

elasticsearch nest elasticsearch-curator

asked Nov 15 '18 at 13:13

DdRegalo

405

asked Nov 15 '18 at 13:13

DdRegalo

405

asked Nov 15 '18 at 13:13

DdRegalo

405

asked Nov 15 '18 at 13:13

DdRegalo

405

asked Nov 15 '18 at 13:13

DdRegalo

405

Can you elaborate the rollover part please ? Dont think I get it. Can’t you just schedule curator to run after rollover ?

– ben5556
Nov 15 '18 at 19:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg: PUT /logs-000001 { "aliases": { "logs_write": {} } } # Add > 1000 documents to logs-000001 POST /logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000, "max_size": "5gb" } } Now, using the curator I intended to config a delete action that deletes all indices except the most recent 2 (eg. the newest index created by the rollover and the latest index that was just rolled over). I had a go with a count filter type but not got it yet!

– DdRegalo
Nov 16 '18 at 9:06

add a comment |

Can you elaborate the rollover part please ? Dont think I get it. Can’t you just schedule curator to run after rollover ?

– ben5556
Nov 15 '18 at 19:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg: PUT /logs-000001 { "aliases": { "logs_write": {} } } # Add > 1000 documents to logs-000001 POST /logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000, "max_size": "5gb" } } Now, using the curator I intended to config a delete action that deletes all indices except the most recent 2 (eg. the newest index created by the rollover and the latest index that was just rolled over). I had a go with a count filter type but not got it yet!

– DdRegalo
Nov 16 '18 at 9:06

Can you elaborate the rollover part please ? Dont think I get it. Can’t you just schedule curator to run after rollover ?

– ben5556
Nov 15 '18 at 19:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg:

PUT /logs-000001  {   "aliases": {     "logs_write": {}   } }  # Add > 1000 documents to logs-000001  POST /logs_write/_rollover  {   "conditions": {     "max_age":   "7d",     "max_docs":  1000,     "max_size":  "5gb"   } }

Now, using the curator I intended to config a delete action that deletes all indices except the most recent 2 (eg. the newest index created by the rollover and the latest index that was just rolled over). I had a go with a count filter type but not got it yet!

– DdRegalo
Nov 16 '18 at 9:06

@ben5556 I've done the rollover in the code using NEST for .NET using aliases => Eg:

PUT /logs-000001  {   "aliases": {     "logs_write": {}   } }  # Add > 1000 documents to logs-000001  POST /logs_write/_rollover  {   "conditions": {     "max_age":   "7d",     "max_docs":  1000,     "max_size":  "5gb"   } }

add a comment |

1 Answer
1

active

oldest

votes

It's actually easier than you think. You can keep the two most recent indices quite easily with the count filter. The following example incorporates both the rollover action and the delete_indices action immediately after it (I used the conditions you supplied in the comment above--tune your rollover conditions appropriately):

actions:

  1:

    action: rollover

    description: Rollover index associated with alias name

    options:

      name: aliasname

      conditions:

        max_age: 7d

        max_docs: 1000

        max_size: 5gb

  2:

    action: delete_indices

    description: Keep only the two most recent indices

    options:

      ignore_empty_list: true

    filters:

    - filtertype: pattern

      kind: prefix

      value: applogging-test

    - filtertype: count

      count: 2

Now, this presupposes that all indices matching prefix applogging-test will be rollover-style, and will increment numerically. You can add other options or filters as needed, though.

answered Nov 17 '18 at 17:35

untergeek

64838

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53320317%2felasticsearch-curator-delete-action-delete-n-older-indices-after-rollover%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

actions:

  1:

    action: rollover

    description: Rollover index associated with alias name

    options:

      name: aliasname

      conditions:

        max_age: 7d

        max_docs: 1000

        max_size: 5gb

  2:

    action: delete_indices

    description: Keep only the two most recent indices

    options:

      ignore_empty_list: true

    filters:

    - filtertype: pattern

      kind: prefix

      value: applogging-test

    - filtertype: count

      count: 2

Now, this presupposes that all indices matching prefix applogging-test will be rollover-style, and will increment numerically. You can add other options or filters as needed, though.

answered Nov 17 '18 at 17:35

untergeek

64838

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

add a comment |

actions:

  1:

    action: rollover

    description: Rollover index associated with alias name

    options:

      name: aliasname

      conditions:

        max_age: 7d

        max_docs: 1000

        max_size: 5gb

  2:

    action: delete_indices

    description: Keep only the two most recent indices

    options:

      ignore_empty_list: true

    filters:

    - filtertype: pattern

      kind: prefix

      value: applogging-test

    - filtertype: count

      count: 2

Now, this presupposes that all indices matching prefix applogging-test will be rollover-style, and will increment numerically. You can add other options or filters as needed, though.

answered Nov 17 '18 at 17:35

untergeek

64838

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

add a comment |

actions:

  1:

    action: rollover

    description: Rollover index associated with alias name

    options:

      name: aliasname

      conditions:

        max_age: 7d

        max_docs: 1000

        max_size: 5gb

  2:

    action: delete_indices

    description: Keep only the two most recent indices

    options:

      ignore_empty_list: true

    filters:

    - filtertype: pattern

      kind: prefix

      value: applogging-test

    - filtertype: count

      count: 2

Now, this presupposes that all indices matching prefix applogging-test will be rollover-style, and will increment numerically. You can add other options or filters as needed, though.

answered Nov 17 '18 at 17:35

untergeek

64838

actions:

  1:

    action: rollover

    description: Rollover index associated with alias name

    options:

      name: aliasname

      conditions:

        max_age: 7d

        max_docs: 1000

        max_size: 5gb

  2:

    action: delete_indices

    description: Keep only the two most recent indices

    options:

      ignore_empty_list: true

    filters:

    - filtertype: pattern

      kind: prefix

      value: applogging-test

    - filtertype: count

      count: 2

Now, this presupposes that all indices matching prefix applogging-test will be rollover-style, and will increment numerically. You can add other options or filters as needed, though.

answered Nov 17 '18 at 17:35

untergeek

64838

answered Nov 17 '18 at 17:35

untergeek

64838

answered Nov 17 '18 at 17:35

untergeek

64838

answered Nov 17 '18 at 17:35

untergeek

64838

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

add a comment |

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

You could run this action file frequently, with no danger of indices rolling over more often than your conditions provide for. Likewise, no deletions would take place unless there were more than 2 indices matching the applogging-test prefix. You could schedule it to run every 5 minutes without concern.

– untergeek
Nov 17 '18 at 17:37

I totally get this...your solution (exactly as you've detailed here with the condition parameter changes) is working like a charm, thanks again! I can also confirm that scheduling a frequent rollover and deletion has no consequence unless the conditions are met in either which is great :)

– DdRegalo
Nov 19 '18 at 14:35

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Vfrdtyky