Taking out items from XML C# and performing bulk delete












1















I have a piece of code which looks like this:



string filePath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + @"DeletedItems" + "DeletedItems" + ".xml";
XDocument xmlDoc = XDocument.Load(filePath);
var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
var idsList = FormItemIdList(list);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);


This part of the code loads up all items from the XML file:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();


Where the XML file looks like this:



<?xml version="1.0" encoding="utf-8"?>
<ItemsToDelete>
<ItemID>113347292264</ItemID>
<ItemID>113334066205</ItemID>
<ItemID>113331816848</ItemID>
<ItemID>113191634415</ItemID>
<ItemID>183480362055</ItemID>
<ItemID>113303425739</ItemID>
<ItemID>112533425202</ItemID>
<ItemID>112007496785</ItemID>
<ItemID>111956371906</ItemID>
<ItemID>112016647700</ItemID>
</ItemsToDelete>


Once the items are loaded into list from C# I form a string which looks like following:



113347292264,113334066205... etc etc


The string is then passed to the stored procedure which looks like following:



create procedure [dbo].[zsp_deleteEndedItems]
(
@ItemIDList nvarchar(max)
)
as
delete from
SearchedUserItems
WHERE EXISTS (SELECT 1 FROM dbo.SplitStringProduction(@ItemIDList,',') S1 WHERE ItemID=S1.val)


Then the SplitStringProduction function is called to break down the passed string and delete the records in DB based on the passed item ID:



create FUNCTION [dbo].[SplitStringProduction]
(
@string nvarchar(max),
@delimiter nvarchar(5)
) RETURNS @t TABLE
(
val nvarchar(500)
)
AS
BEGIN
declare @xml xml
set @xml = N'<root><r>' + replace(@string,@delimiter,'</r><r>') + '</r></root>'

insert into @t(val)
select
r.value('.','varchar(500)') as item
from @xml.nodes('//root/r') as records(r)

RETURN
END


Now, all this works fine, but there are some problems that I'm aware of and that I would like to avoid:




  1. The list from the XML file can be very long, often containing 100000 + records inside


Now to avoid this I was thinkng to do the following:




  • Perform deletion in batches of 5000 records let's say


So the steps would be:




  • Pull 5000 items from the XML file

  • Pass those 5000 items into the procedure

  • Remove the 5000 ItemID's from the XML file


Can someone help me out with this, I'm not sure how to do it ?










share|improve this question

























  • There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

    – mjwills
    Nov 15 '18 at 11:11













  • @mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

    – User987
    Nov 15 '18 at 11:21











  • If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

    – H.Mikhaeljan
    Nov 15 '18 at 11:27
















1















I have a piece of code which looks like this:



string filePath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + @"DeletedItems" + "DeletedItems" + ".xml";
XDocument xmlDoc = XDocument.Load(filePath);
var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
var idsList = FormItemIdList(list);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);


This part of the code loads up all items from the XML file:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();


Where the XML file looks like this:



<?xml version="1.0" encoding="utf-8"?>
<ItemsToDelete>
<ItemID>113347292264</ItemID>
<ItemID>113334066205</ItemID>
<ItemID>113331816848</ItemID>
<ItemID>113191634415</ItemID>
<ItemID>183480362055</ItemID>
<ItemID>113303425739</ItemID>
<ItemID>112533425202</ItemID>
<ItemID>112007496785</ItemID>
<ItemID>111956371906</ItemID>
<ItemID>112016647700</ItemID>
</ItemsToDelete>


Once the items are loaded into list from C# I form a string which looks like following:



113347292264,113334066205... etc etc


The string is then passed to the stored procedure which looks like following:



create procedure [dbo].[zsp_deleteEndedItems]
(
@ItemIDList nvarchar(max)
)
as
delete from
SearchedUserItems
WHERE EXISTS (SELECT 1 FROM dbo.SplitStringProduction(@ItemIDList,',') S1 WHERE ItemID=S1.val)


Then the SplitStringProduction function is called to break down the passed string and delete the records in DB based on the passed item ID:



create FUNCTION [dbo].[SplitStringProduction]
(
@string nvarchar(max),
@delimiter nvarchar(5)
) RETURNS @t TABLE
(
val nvarchar(500)
)
AS
BEGIN
declare @xml xml
set @xml = N'<root><r>' + replace(@string,@delimiter,'</r><r>') + '</r></root>'

insert into @t(val)
select
r.value('.','varchar(500)') as item
from @xml.nodes('//root/r') as records(r)

RETURN
END


Now, all this works fine, but there are some problems that I'm aware of and that I would like to avoid:




  1. The list from the XML file can be very long, often containing 100000 + records inside


Now to avoid this I was thinkng to do the following:




  • Perform deletion in batches of 5000 records let's say


So the steps would be:




  • Pull 5000 items from the XML file

  • Pass those 5000 items into the procedure

  • Remove the 5000 ItemID's from the XML file


Can someone help me out with this, I'm not sure how to do it ?










share|improve this question

























  • There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

    – mjwills
    Nov 15 '18 at 11:11













  • @mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

    – User987
    Nov 15 '18 at 11:21











  • If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

    – H.Mikhaeljan
    Nov 15 '18 at 11:27














1












1








1


0






I have a piece of code which looks like this:



string filePath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + @"DeletedItems" + "DeletedItems" + ".xml";
XDocument xmlDoc = XDocument.Load(filePath);
var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
var idsList = FormItemIdList(list);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);


This part of the code loads up all items from the XML file:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();


Where the XML file looks like this:



<?xml version="1.0" encoding="utf-8"?>
<ItemsToDelete>
<ItemID>113347292264</ItemID>
<ItemID>113334066205</ItemID>
<ItemID>113331816848</ItemID>
<ItemID>113191634415</ItemID>
<ItemID>183480362055</ItemID>
<ItemID>113303425739</ItemID>
<ItemID>112533425202</ItemID>
<ItemID>112007496785</ItemID>
<ItemID>111956371906</ItemID>
<ItemID>112016647700</ItemID>
</ItemsToDelete>


Once the items are loaded into list from C# I form a string which looks like following:



113347292264,113334066205... etc etc


The string is then passed to the stored procedure which looks like following:



create procedure [dbo].[zsp_deleteEndedItems]
(
@ItemIDList nvarchar(max)
)
as
delete from
SearchedUserItems
WHERE EXISTS (SELECT 1 FROM dbo.SplitStringProduction(@ItemIDList,',') S1 WHERE ItemID=S1.val)


Then the SplitStringProduction function is called to break down the passed string and delete the records in DB based on the passed item ID:



create FUNCTION [dbo].[SplitStringProduction]
(
@string nvarchar(max),
@delimiter nvarchar(5)
) RETURNS @t TABLE
(
val nvarchar(500)
)
AS
BEGIN
declare @xml xml
set @xml = N'<root><r>' + replace(@string,@delimiter,'</r><r>') + '</r></root>'

insert into @t(val)
select
r.value('.','varchar(500)') as item
from @xml.nodes('//root/r') as records(r)

RETURN
END


Now, all this works fine, but there are some problems that I'm aware of and that I would like to avoid:




  1. The list from the XML file can be very long, often containing 100000 + records inside


Now to avoid this I was thinkng to do the following:




  • Perform deletion in batches of 5000 records let's say


So the steps would be:




  • Pull 5000 items from the XML file

  • Pass those 5000 items into the procedure

  • Remove the 5000 ItemID's from the XML file


Can someone help me out with this, I'm not sure how to do it ?










share|improve this question
















I have a piece of code which looks like this:



string filePath = Environment.GetFolderPath(Environment.SpecialFolder.Desktop) + @"DeletedItems" + "DeletedItems" + ".xml";
XDocument xmlDoc = XDocument.Load(filePath);
var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
var idsList = FormItemIdList(list);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);


This part of the code loads up all items from the XML file:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();


Where the XML file looks like this:



<?xml version="1.0" encoding="utf-8"?>
<ItemsToDelete>
<ItemID>113347292264</ItemID>
<ItemID>113334066205</ItemID>
<ItemID>113331816848</ItemID>
<ItemID>113191634415</ItemID>
<ItemID>183480362055</ItemID>
<ItemID>113303425739</ItemID>
<ItemID>112533425202</ItemID>
<ItemID>112007496785</ItemID>
<ItemID>111956371906</ItemID>
<ItemID>112016647700</ItemID>
</ItemsToDelete>


Once the items are loaded into list from C# I form a string which looks like following:



113347292264,113334066205... etc etc


The string is then passed to the stored procedure which looks like following:



create procedure [dbo].[zsp_deleteEndedItems]
(
@ItemIDList nvarchar(max)
)
as
delete from
SearchedUserItems
WHERE EXISTS (SELECT 1 FROM dbo.SplitStringProduction(@ItemIDList,',') S1 WHERE ItemID=S1.val)


Then the SplitStringProduction function is called to break down the passed string and delete the records in DB based on the passed item ID:



create FUNCTION [dbo].[SplitStringProduction]
(
@string nvarchar(max),
@delimiter nvarchar(5)
) RETURNS @t TABLE
(
val nvarchar(500)
)
AS
BEGIN
declare @xml xml
set @xml = N'<root><r>' + replace(@string,@delimiter,'</r><r>') + '</r></root>'

insert into @t(val)
select
r.value('.','varchar(500)') as item
from @xml.nodes('//root/r') as records(r)

RETURN
END


Now, all this works fine, but there are some problems that I'm aware of and that I would like to avoid:




  1. The list from the XML file can be very long, often containing 100000 + records inside


Now to avoid this I was thinkng to do the following:




  • Perform deletion in batches of 5000 records let's say


So the steps would be:




  • Pull 5000 items from the XML file

  • Pass those 5000 items into the procedure

  • Remove the 5000 ItemID's from the XML file


Can someone help me out with this, I'm not sure how to do it ?







c# asp.net linq stored-procedures c#-4.0






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 15 '18 at 11:20







User987

















asked Nov 15 '18 at 10:57









User987User987

1,35932147




1,35932147













  • There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

    – mjwills
    Nov 15 '18 at 11:11













  • @mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

    – User987
    Nov 15 '18 at 11:21











  • If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

    – H.Mikhaeljan
    Nov 15 '18 at 11:27



















  • There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

    – mjwills
    Nov 15 '18 at 11:11













  • @mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

    – User987
    Nov 15 '18 at 11:21











  • If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

    – H.Mikhaeljan
    Nov 15 '18 at 11:27

















There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

– mjwills
Nov 15 '18 at 11:11







There is a lot of text there, and questions like But I'm not sure what is the best way to do it here? is likely to result in being closed as 'primarily opinion-based'. It might help if you simplify the question a little.

– mjwills
Nov 15 '18 at 11:11















@mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

– User987
Nov 15 '18 at 11:21





@mjwills I'd simply want to code out those steps at the end that I wrote.. that's pretty much it

– User987
Nov 15 '18 at 11:21













If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

– H.Mikhaeljan
Nov 15 '18 at 11:27





If the first step is not an issue then you can use this to create batches of 5000 of the current list and then loop through the lists and send them in. Else you have to use XmlReader to not load everything into memory and do some more work but the idea would be the same.

– H.Mikhaeljan
Nov 15 '18 at 11:27












1 Answer
1






active

oldest

votes


















1














It sounds like you just want to split the list into smaller chunks:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
while(list.Any())
{
var subList = list.Take(5000);
var idsList = FormItemIdList(subList);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);

list.RemoveRange(subList);
}





share|improve this answer
























  • Haha this is exactly it :)

    – User987
    Nov 15 '18 at 11:29











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53317915%2ftaking-out-items-from-xml-c-sharp-and-performing-bulk-delete%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














It sounds like you just want to split the list into smaller chunks:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
while(list.Any())
{
var subList = list.Take(5000);
var idsList = FormItemIdList(subList);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);

list.RemoveRange(subList);
}





share|improve this answer
























  • Haha this is exactly it :)

    – User987
    Nov 15 '18 at 11:29
















1














It sounds like you just want to split the list into smaller chunks:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
while(list.Any())
{
var subList = list.Take(5000);
var idsList = FormItemIdList(subList);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);

list.RemoveRange(subList);
}





share|improve this answer
























  • Haha this is exactly it :)

    – User987
    Nov 15 '18 at 11:29














1












1








1







It sounds like you just want to split the list into smaller chunks:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
while(list.Any())
{
var subList = list.Take(5000);
var idsList = FormItemIdList(subList);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);

list.RemoveRange(subList);
}





share|improve this answer













It sounds like you just want to split the list into smaller chunks:



var list = xmlDoc.Root.Elements("ItemID")
.Select(element => element.Value)
.ToList();
while(list.Any())
{
var subList = list.Take(5000);
var idsList = FormItemIdList(subList);

ctx.zsp_deleteEndedItems(idsList);
ctx.zsp_deleteEndedItemsTransactions(idsList);

list.RemoveRange(subList);
}






share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 15 '18 at 11:28









NeilNeil

4,85711537




4,85711537













  • Haha this is exactly it :)

    – User987
    Nov 15 '18 at 11:29



















  • Haha this is exactly it :)

    – User987
    Nov 15 '18 at 11:29

















Haha this is exactly it :)

– User987
Nov 15 '18 at 11:29





Haha this is exactly it :)

– User987
Nov 15 '18 at 11:29




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53317915%2ftaking-out-items-from-xml-c-sharp-and-performing-bulk-delete%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Xamarin.iOS Cant Deploy on Iphone

Glorious Revolution

Dulmage-Mendelsohn matrix decomposition in Python