TensorFlow Implementing a new optimizer: Computing the sum of slot variable
I am rather new to tensorflow and I am struggling to implement a new optimizer that I would like to test. I think that most of the problems I am facing are related to the poor knowledge of tensor flow and the way tensors are handled.
This is what I need to do: My optimizer has some parameters that should be updated on runtime. The way the parameters should be updated is by computing the sum of all the the components of a variable of the optimizer. If I understood correctly the variables of my optimizer should be defined by using "slot_variables" inside the _create_slots(self, var_list) method. This is something like:
for v in var_list:
self._zeros_slot(v, "p_t", self._name)
Then the parameter that I should update as a function of the sum, should be declared as a non slot variable:
first_var = min(var_list, key=lambda x: x.name)
self._create_non_slot_variable(initial_value=0., name="p", colocate_with=first_var)
First of all, is what I wrote above correct?
If so, I tried to update my parameters in the _finish(self, update_ops, name_scope) method:
list_vars = tf.trainable_variables()
p.assign(0) # This is the sum of the variable i want to compute
for var in list_vars:
p_t = self.get_slot(var, "p_t")
p.assign_add(tf.reduce_sum(p_t))
with ops.colocate_with(p):
# negative_moment and positive_moment are two methods if call depending on the result of the sum of the variables.
out = tf.cond(tf.less(p, 0), negative_moment, possitive_moment)
Is the strategy that I am trying to use correct?
Just for information, when I try to use my optimizer to minimize the Rosenbrock function, it works properly, but it does not work when I try to use on MNIST. I suspect there is a conceptual problem on my implementation but I don't know where the problem could be.
I thank you all in advance for your help.
Cheers,
Daniel
python tensorflow
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
add a comment |
I am rather new to tensorflow and I am struggling to implement a new optimizer that I would like to test. I think that most of the problems I am facing are related to the poor knowledge of tensor flow and the way tensors are handled.
This is what I need to do: My optimizer has some parameters that should be updated on runtime. The way the parameters should be updated is by computing the sum of all the the components of a variable of the optimizer. If I understood correctly the variables of my optimizer should be defined by using "slot_variables" inside the _create_slots(self, var_list) method. This is something like:
for v in var_list:
self._zeros_slot(v, "p_t", self._name)
Then the parameter that I should update as a function of the sum, should be declared as a non slot variable:
first_var = min(var_list, key=lambda x: x.name)
self._create_non_slot_variable(initial_value=0., name="p", colocate_with=first_var)
First of all, is what I wrote above correct?
If so, I tried to update my parameters in the _finish(self, update_ops, name_scope) method:
list_vars = tf.trainable_variables()
p.assign(0) # This is the sum of the variable i want to compute
for var in list_vars:
p_t = self.get_slot(var, "p_t")
p.assign_add(tf.reduce_sum(p_t))
with ops.colocate_with(p):
# negative_moment and positive_moment are two methods if call depending on the result of the sum of the variables.
out = tf.cond(tf.less(p, 0), negative_moment, possitive_moment)
Is the strategy that I am trying to use correct?
Just for information, when I try to use my optimizer to minimize the Rosenbrock function, it works properly, but it does not work when I try to use on MNIST. I suspect there is a conceptual problem on my implementation but I don't know where the problem could be.
I thank you all in advance for your help.
Cheers,
Daniel
python tensorflow
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
add a comment |
I am rather new to tensorflow and I am struggling to implement a new optimizer that I would like to test. I think that most of the problems I am facing are related to the poor knowledge of tensor flow and the way tensors are handled.
This is what I need to do: My optimizer has some parameters that should be updated on runtime. The way the parameters should be updated is by computing the sum of all the the components of a variable of the optimizer. If I understood correctly the variables of my optimizer should be defined by using "slot_variables" inside the _create_slots(self, var_list) method. This is something like:
for v in var_list:
self._zeros_slot(v, "p_t", self._name)
Then the parameter that I should update as a function of the sum, should be declared as a non slot variable:
first_var = min(var_list, key=lambda x: x.name)
self._create_non_slot_variable(initial_value=0., name="p", colocate_with=first_var)
First of all, is what I wrote above correct?
If so, I tried to update my parameters in the _finish(self, update_ops, name_scope) method:
list_vars = tf.trainable_variables()
p.assign(0) # This is the sum of the variable i want to compute
for var in list_vars:
p_t = self.get_slot(var, "p_t")
p.assign_add(tf.reduce_sum(p_t))
with ops.colocate_with(p):
# negative_moment and positive_moment are two methods if call depending on the result of the sum of the variables.
out = tf.cond(tf.less(p, 0), negative_moment, possitive_moment)
Is the strategy that I am trying to use correct?
Just for information, when I try to use my optimizer to minimize the Rosenbrock function, it works properly, but it does not work when I try to use on MNIST. I suspect there is a conceptual problem on my implementation but I don't know where the problem could be.
I thank you all in advance for your help.
Cheers,
Daniel
python tensorflow
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
I am rather new to tensorflow and I am struggling to implement a new optimizer that I would like to test. I think that most of the problems I am facing are related to the poor knowledge of tensor flow and the way tensors are handled.
This is what I need to do: My optimizer has some parameters that should be updated on runtime. The way the parameters should be updated is by computing the sum of all the the components of a variable of the optimizer. If I understood correctly the variables of my optimizer should be defined by using "slot_variables" inside the _create_slots(self, var_list) method. This is something like:
for v in var_list:
self._zeros_slot(v, "p_t", self._name)
Then the parameter that I should update as a function of the sum, should be declared as a non slot variable:
first_var = min(var_list, key=lambda x: x.name)
self._create_non_slot_variable(initial_value=0., name="p", colocate_with=first_var)
First of all, is what I wrote above correct?
If so, I tried to update my parameters in the _finish(self, update_ops, name_scope) method:
list_vars = tf.trainable_variables()
p.assign(0) # This is the sum of the variable i want to compute
for var in list_vars:
p_t = self.get_slot(var, "p_t")
p.assign_add(tf.reduce_sum(p_t))
with ops.colocate_with(p):
# negative_moment and positive_moment are two methods if call depending on the result of the sum of the variables.
out = tf.cond(tf.less(p, 0), negative_moment, possitive_moment)
Is the strategy that I am trying to use correct?
Just for information, when I try to use my optimizer to minimize the Rosenbrock function, it works properly, but it does not work when I try to use on MNIST. I suspect there is a conceptual problem on my implementation but I don't know where the problem could be.
I thank you all in advance for your help.
Cheers,
Daniel
python tensorflow
python tensorflow
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
asked Nov 15 '18 at 13:27
alpinito1alpinito1
11
11
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53320554%2ftensorflow-implementing-a-new-optimizer-computing-the-sum-of-slot-variable%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53320554%2ftensorflow-implementing-a-new-optimizer-computing-the-sum-of-slot-variable%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
