Vfrdtyky

I need to assign different values of memory for each new worker. So I tried changing memory for each bolt and spout. I am currently using a custom scheduler also. Here is my approach to the problem.

MY CODE:

TopologyBuilder builder = new TopologyBuilder();

builder.setSpout("spout", new EmailSpout(), 1).addConfiguration("node", "zoo1").setMemoryLoad(512.0);

builder.setBolt("increment1", new IncrementBolt(), PARALLELISM).shuffleGrouping("spout").addConfiguration("node", "zoo2").setMemoryLoad(2048.0);

builder.setBolt("increment2", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment1").addConfiguration("node", "zoo3").setMemoryLoad(2048.0);

builder.setBolt("increment3", new IncrementBolt(), PARALLELISM).shuffleGrouping("increment2").addConfiguration("node", "zoo4").setMemoryLoad(2048.0);

builder.setBolt("output", new OutputBolt(), 1).globalGrouping("increment2").addConfiguration("node", "zoo1").setMemoryLoad(512.0);

Config conf = new Config();

conf.setDebug(false);

conf.setNumWorkers(4);

StormSubmitter.submitTopologyWithProgressBar("Microbenchmark", conf, builder.createTopology());

MY STORM.YAML:

 storm.zookeeper.servers:

     - "zoo1"

 storm.zookeeper.port: 2181

 nimbus.seeds: ["zoo1"]

 storm.local.dir: "/home/ubuntu/eranga/storm-data"

 supervisor.slots.ports:

     - 6700

     - 6701

     - 6702

     - 6703

     - 6704

 storm.scheduler: "org.apache.storm.scheduler.NodeBasedCustomScheduler"

 supervisor.scheduler.meta:

     node: "zoo4"

 worker.profiler.enabled: true

 worker.profiler.childopts: "-XX:+UnlockCommercialFeatures -XX:+FlightRecorder"

 worker.profiler.command: "flight.bash"

 worker.heartbeat.frequency.secs: 1

 worker.childopts: "-Xmx2048m -Xms2048m -Djava.net.preferIPv4Stack=true -Dorg.xml.sax.driver=com.sun.org.apache.xerces.internal.parsers.SAXParser -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"

When I submit the topology I get the following error.

ERROR:

Exception in thread "main" java.lang.IllegalArgumentException: Topology will not be able to be successfully scheduled: Config TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB=768.0 < 2048.0 (Largest memory requirement of a component in the topology). Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount

at org.apache.storm.StormSubmitter.validateTopologyWorkerMaxHeapSizeMBConfigs(StormSubmitter.java:496)

Any suggestions?

Did you try following the advice from the error message?

Perhaps set TOPOLOGY_WORKER_MAX_HEAP_SIZE_MB to a larger amount

Try adding this to storm.yaml:

topology.worker.max.heap.size.mb=2048.0 

Try using this.

import org.apache.storm.Config;



public class TopologyExecuter{

   for(List<StormTopology> StormTopologyObject : StormTopologyObjects ){

       Config topologyConf = new Config();

       topologyConf.put(Config.TOPOLOGY_WORKER_CHILDOPTS,"-Xmx512m -Xms256m");

       StormSubmitter.submitTopology("topology name", topologyConf, StormTopologyObject);

   }



}