Cassandra Administration and Security

nodetool

To manage Cassandra with nodetool

bin/nodetool -h 10.10.10.1 [-p JMX_PORT ] Command...

Pass in credential if required

bin/nodetool -h 10.10.10.1 [-p JMX_PORT -u JMX_USERNAME -p JMX_PASSWORD ] Command...

Cassandra nodetool Command

usage: java org.apache.cassandra.tools.NodeCmd --host <arg> <command>

 -h,--host <arg>        node hostname or ip address
 -p,--port <arg>        remote jmx agent port number
 -pw,--password <arg>   remote jmx agent password
 -u,--username <arg>    remote jmx agent username

Available commands:
  ring                   - Print informations on the token ring
  join                   - Join the ring
  info                   - Print node informations (uptime, load, ...)
  cfstats                - Print statistics on column families
  clearsnapshot          - Remove all existing snapshots
  version                - Print cassandra version
  tpstats                - Print usage statistics of thread pools
  drain                  - Drain the node (stop accepting writes and flush all column families)
  decommission           - Decommission the node
  loadbalance            - Loadbalance the node
  compactionstats        - Print statistics on compactions
  disablegossip          - Disable gossip (effectively marking the node dead)
  enablegossip           - Reenable gossip
  disablethrift          - Disable thrift server
  enablethrift           - Reenable thrift server
  snapshot [snapshotname] - Take a snapshot using optional name snapshotname
  netstats [host]        - Print network information on provided host (connecting node by default)
  move <new token>       - Move node on the token ring to a new token
  removetoken status|force|<token> - Show status of current token removal, force completion of pending removal or remove providen token
  flush [keyspace] [cfnames] - Flush one or more column family
  repair [keyspace] [cfnames] - Repair one or more column family
  cleanup [keyspace] [cfnames] - Run cleanup on one or more column family
  compact [keyspace] [cfnames] - Force a (major) compaction on one or more column family
  scrub [keyspace] [cfnames] - Scrub (rebuild sstables for) one or more column family
  invalidatekeycache [keyspace] [cfnames] - Invalidate the key cache of one or more column family
  invalidaterowcache [keyspace] [cfnames] - Invalidate the key cache of one or more column family
  getcompactionthreshold <keyspace> <cfname> - Print min and max compaction thresholds for a given column family
  cfhistograms <keyspace> <cfname> - Print statistic histograms for a given column family
  setcachecapacity <keyspace> <cfname> <keycachecapacity> <rowcachecapacity> - Set the key and row cache capacities of a given column family
  setcompactionthreshold <keyspace> <cfname> <minthreshold> <maxthreshold> - Set the min and max compaction thresholds for a given column family
  • nodetool ring
    • Display ring & node status from the node it queried
    • Display any nodes are dead
    • Check if loads are balanced (disk storage)
    • If information is in-consistence from different queried nodes, the ring configuration may be wrong.
      bin/nodetool -h 10.10.10.1 info
      Address         Status State   Load            Owns    Token
                                                             163572425264069043502692069140600439631
      10.10.10.1   Up     Normal  111.57 KB       70.70%     113716211212737963740265714504910561460
      10.10.10.2   Up     Normal  111.76 KB       29.30%     163572425264069043502692069140600439631
  • nodetool join
    • Join a new node to a ring (Verify seed list, initial token and auto bootstrapping values before joining a node)
  • nodetool info
    • Output node information
      bin/nodetool -h 10.208.115.203 info
      163572425264069043502692069140600439631
      Gossip active    : true
      Load             : 111.76 KB
      Generation No    : 1304619658
      Uptime (seconds) : 23993
      Heap Memory (MB) : 48.19 / 290.63
  • nodetool cfstats
    • Print information on every keyspace and column family (cf)
      ----------------
      Keyspace: store
              Read Count: 0
              Read Latency: NaN ms.
              Write Count: 0
              Write Latency: NaN ms.
              Pending Tasks: 0
                      Column Family: products
                      SSTable count: 0
                      Space used (live): 0
                      Space used (total): 0
                      Memtable Columns Count: 0
                      Memtable Data Size: 0
                      Memtable Switch Count: 0
                      Read Count: 0
                      Read Latency: NaN ms.
                      Write Count: 0
                      Write Latency: NaN ms.
                      Pending Tasks: 0
                      Key cache capacity: 200000
                      Key cache size: 0
                      Key cache hit rate: NaN
                      Row cache: disabled
                      Compacted row minimum size: 0
                      Compacted row maximum size: 0
                      Compacted row mean size: 0
      
                      Column Family: users
                      SSTable count: 1
                      Space used (live): 4518
                      Space used (total): 4518
                      Memtable Columns Count: 0
                      Memtable Data Size: 0
                      Memtable Switch Count: 0
                      Read Count: 0
                      Read Latency: NaN ms.
                      Write Count: 0
                      Write Latency: NaN ms.
                      Pending Tasks: 0
                      Key cache capacity: 200000
                      Key cache size: 2
                      Key cache hit rate: NaN
                      Row cache: disabled
                      Compacted row minimum size: 87
                      Compacted row maximum size: 124
                      Compacted row mean size: 114
      
      
      ----------------
      Keyspace: order
      ...
  • nodetool version
    • Print Cassandra version
  • nodetool cleanup [my_keyspace] [my_column_family]
    • Clean up keys no longer belonging to this node
  • nodetool compact [my_keyspace] [my_column_family]
    • Perform a major compaction - Initiates an immediate major compaction of all column families in keyspace. Compacts all column family's SSTables into a single SSTable
  • nodetool cfhistograms my_keyspace my_column_family
    • Print statistics on the read/write latency on a column family
      Offset      SSTables     Write Latency      Read Latency          Row Size      Column Count
      1                  0                 0                 0                 0                 1
      2                  0                 0                 0                 0                 1
  • nodetool snapshot [my_snapshot_name]
    • Data is flushed and then take a snapshot all Cassandra data
    • Snapshot is stored under
      /var/lib/cassandra/data/mykeyspace/snapshots/timestamp-my_snapshot_name
  • nodetool clearsnapshot
    • Deletes all snapshots
  • nodetool tpstats
    • Print all tasks status
      Pool Name                    Active   Pending      Completed
      ReadStage                         0         0              2
      RequestResponseStage              0         0              0
      MutationStage                     0         0              3
      ReadRepairStage                   0         0              0
      GossipStage                       0         0          74980
      AntiEntropyStage                  0         0              0
      MigrationStage                    0         0              0
      MemtablePostFlusher               0         0              2
      StreamStage                       0         0              0
      FlushWriter                       0         0              2
      MiscStage                         0         0              0
      FlushSorter                       0         0              0
      InternalResponseStage             0         0              0
      HintedHandoff                     0         0              1
  • nodetool flush my_keyspace [my_column_family]
    • Flushes memtables for a keyspace to disk
    • Clear the commit log
  • nodetool drain
    • Flushes memtables for a node
    • Stop write operation (Read will continue to function)
    • Perform to upgrade a node to a new version of Cassandra
  • nodetool repair keyspace [my_column_family]
    • Start AntiEntropy repair
    • All data for a keyspace will be compared among all replicas
    • All replica date for a keyspace will be updated to the most updated one
  • nodetool decommission
    • Decommission a live node and migrate the data to its next node
  • nodetool move new_token
    • Move a server responsible for a new token and hence will responsible for a different range of token keys
    • It involves decommission and bootstrap a node
  • nodetool netstats host
    • Displays network information
      Mode: Normal
       Nothing streaming to /10.10.10.2
       Nothing streaming from /10.10.10.2
      Pool Name                    Active   Pending      Completed
      Commands                        n/a         0              1
      Responses                       n/a         0          77034
  • nodetool removetoken status | force | token
    • status: shows status of a token removal
    • force: forces the the completion of a removal,
    • token: removes a specified token
    • The next node will resume responsibility to read and replicate requests
  • nodetool setcachecapacity keyspace mycolumn_family key_cache_capacity row_cache_capacity
    • Set the size of the key cache and row cache
  • nodetool invalidatekeycache [my_keyspace] [my_column_family]
    • Invalidates or deletes the key cache
  • nodetool invalidaterowcache [my_keyspace] [my_column_family]
    • Invalidates or deletes the row cache
  • nodetool getcompactionthreshold keyspace my_column_family
    • Gets the current compaction threshold
      Current compaction thresholds for store/users:
       min = 4,  max = 32
  • nodetool setcompactionthreshold my_column_family min_threshold max_threshold
    • Set compaction thresholds

Cassandra Authentication & Authorization

  • Edit cassandra.yaml to enable authentication
    authenticator: org.apache.cassandra.auth.SimpleAuthenticator
  • Edit access.properties for individual user read and write privileges
    MyKeySpace[.MyColumnFamily].PERMISSION=MyUsers
    # Right to modify list of keyspaces
    <modify-keyspaces>=jsmith
    
    # Access to Keyspace1
    Keyspace1.<ro>=jsmith,Elvis Presley
    Keyspace1.<rw>=dilbert
    
    # Access to Column Family Standard1 of Keyspace1
    Keyspace1.Standard1.<rw>=jsmith,Elvis Presley,dilbert
    • ro: read onlly
    • rw: read and write privilege
  • Edit the password file passwd.properties for users password
    jsmith=havebadpass
    • To enable MD5 for password encryption
      conf/cassandra-env.sh
      JVM_OPTS="$JVM_OPTS -Dpasswd.mode=MD5"
  • Restart cassandra with the access and password files
    bin/cassandra -f -Dpasswd.properties=conf/passwd.properties -Daccess.properties=conf/access.properties

Logging

Cassandra Log4J logging configuration file is located in

conf/log4j-server.properties