Vfrdtyky

Cassandra cluster specs:

Nodes: 6

Storage: 1536 GB

Cores: 48

Ram: 168 GB

Latency to cassandra cluster from my local: 330 ms-390 ms

I am using cassandra java driver, spark-cassandra-connector_2.11 version 2.3.2

cluster configuration in java driver:

private static Session connect(

      final String node, final Integer port, final String userName, final String password) {



    Builder b = Cluster.builder().addContactPoints(node.split(COMMA_SEPARATOR));

    if (!Strings.isNullOrEmpty(userName) && !Strings.isNullOrEmpty(password)) {

      b.withCredentials(userName, password);

    }



    if (port != null && port != 0) {

      b.withPort(port);

    }



    PoolingOptions poolingOptions = new PoolingOptions();

    poolingOptions

        .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)

        .setMaxRequestsPerConnection(HostDistance.REMOTE, 10000)

        .setMaxConnectionsPerHost(HostDistance.LOCAL, 3)

        .setMaxConnectionsPerHost(HostDistance.REMOTE, 3)

        .setNewConnectionThreshold(HostDistance.LOCAL, 3)

        .setNewConnectionThreshold(HostDistance.REMOTE, 3)

        .setCoreConnectionsPerHost(HostDistance.LOCAL, 3)

        .setCoreConnectionsPerHost(HostDistance.REMOTE, 3);



    b.withSocketOptions(

        new SocketOptions()

            .setConnectTimeoutMillis(SOCKET_CONNECT_TIMEOUT)

            .setReadTimeoutMillis(SOCKET_READ_TIMEOUT));

    b.withPoolingOptions(poolingOptions);



    cluster = b.build();





    session = cluster.connect();



    return session;

  }

Below is my test table:

CREATE TABLE my_keyspace.test_table (

  id int PRIMARY KEY

)

To write to cassandra I am using session.executeAsync and storing the futures in list and waiting for all futures to complete.

When I do 100000 writes it takes 50-65 seconds.
Is it supposed to be this slow or there is something I am missing in configuration?
I have already tried several option in socket options and pooling options, but that's the best I got.

The first thing I would check is whether your Cassandra server is running at 100% CPU utilization. If it isn't, and since I assume that the server is not bottlenecked on disk (no problem to do 1500 writes per second even to a spinning disk), then the bottleneck has to be somewhere else:

One possibility you should always check first is that the client isn't the bottleneck, i.e., it doesn't use 100% CPU.

Then, you said that "Latency to cassandra cluster from my local is 330ms". Is this the ping time between your test machine and the Cassandra cluster? If so, you may have two kinds of problem. First, maybe this is some sort of low-bandwidth WAN, which really can't support more than 2000 requests per second. But I doubt that. Another possibility is that your client simply doesn't have enough concurrency... With 1/3 second latency, to achieve 2000 writes per second, you need the client to do 666 requests in parallel. Is setMaxRequestsPerConnection() you set really taking effect? Because if it isn't, the default (according to https://docs.datastax.com/en/developer/java-driver/2.1/manual/pooling/ ) is 256 times the 3 connections you set, which is 768, close to the above number 666.

And of course it can be many other things. It's hard to guess without more data.