database administration - Cassandra: Removing a node -

i remove node cassandra cluster , following these 2 related questions (here , here) , cassandra document. still not quite sure exact process.

my first question is: following way remove node cassandra cluster correct?

1. decommission node remove.
2. removetoken node decommissioned.

if above process right, how can tell decommission process completed can proceed second step? or safe step 2 right after step 1?

in addition, cassandra document says:

you can take node out of cluster nodetool decommission live node, or nodetool removetoken (to other machine) remove dead one. assign ranges old node responsible other nodes, , replicate appropriate data there. if decommission used, data stream decommissioned node. if removetoken used, data stream remaining replicas.

no data removed automatically node being decommissioned, if want put node service @ different token on ring, should removed manually.

does mean decommissioned node dead node? in addition, no data removed automatically node being decommissioned, how can tell when safe remove data decommissioned node (i.e., how know when data-streaming completed?)

removing node cassandra cluster should following steps (in cassandra v1.2.8):

1. decommission target node nodetool -host node decmmission.
2. once data streaming decommissioned node completed, manually delete data in decommissioned node (optional).

update: above process works seed nodes. in such case, cluster still able run smoothly without requiring restart. when need restart cluster other reasons, sure update seeds parameter specified in cassandra.yaml nodes.

---

decommission target node

when decommission starts, decommissioned node first labeled leaving (marked l). in following example, remove node-76:

> nodetool -host node-76 decommission > nodetool status  status=up/down |/ state=normal/leaving/joining/moving --  address  load       tokens  owns   host id                               rack un  node-70  9.79 gb    256     8.3%   e0a7fb7a-06f8-4f8b-882d-c60bff51328a  155 un  node-80  8.9 gb     256     9.2%   43dfc22e-b838-4b0b-9b20-66a048f73d5f  155 un  node-72  9.47 gb    256     9.2%   75ebf2a9-e83c-4206-9814-3685e5fa0ab5  155 un  node-71  9.48 gb    256     9.5%   cdbfafef-4bfb-4b11-9fb8-27757b0caa47  155 un  node-91  8.05 gb    256     8.4%   6711f8a7-d398-4f93-bd73-47c8325746c3  155 un  node-78  9.11 gb    256     9.4%   c82ace5f-9b90-4f5c-9d86-0fbfb7ac2911  155 ul  node-76  8.36 gb    256     9.5%   15d74e9e-2791-4056-a341-c02f6614b8ae  155 un  node-73  9.36 gb    256     8.9%   c1dfab95-d476-4274-acac-cf6630375566  155 un  node-75  8.93 gb    256     8.2%   8789d89d-2db8-4ddf-bc2d-60ba5edfd0ad  155 un  node-74  8.91 gb    256     9.6%   581fd5bc-20d2-4528-b15d-7475eb2bf5af  155 un  node-79  9.71 gb    256     9.9%   8e192e01-e8eb-4425-9c18-60279b9046ff  155 

when decommissioned node marked leaving, streaming data other living nodes. once streaming completed, node not observed ring structure, , data owned other nodes increase:

status=up/down |/ state=normal/leaving/joining/moving --  address  load       tokens  owns   host id                               rack un  node-70  9.79 gb    256     9.3%   e0a7fb7a-06f8-4f8b-882d-c60bff51328a  155 un  node-80  8.92 gb    256     9.6%   43dfc22e-b838-4b0b-9b20-66a048f73d5f  155 un  node-72  9.47 gb    256     10.2%  75ebf2a9-e83c-4206-9814-3685e5fa0ab5  155 un  node-71  9.69 gb    256     10.6%  cdbfafef-4bfb-4b11-9fb8-27757b0caa47  155 un  node-91  8.05 gb    256     9.1%   6711f8a7-d398-4f93-bd73-47c8325746c3  155 un  node-78  9.11 gb    256     10.5%  c82ace5f-9b90-4f5c-9d86-0fbfb7ac2911  155 un  node-73  9.36 gb    256     9.7%   c1dfab95-d476-4274-acac-cf6630375566  155 un  node-75  9.01 gb    256     9.5%   8789d89d-2db8-4ddf-bc2d-60ba5edfd0ad  155 un  node-74  8.91 gb    256     10.5%  581fd5bc-20d2-4528-b15d-7475eb2bf5af  155 un  node-79  9.71 gb    256     11.0%  8e192e01-e8eb-4425-9c18-60279b9046ff  155 

---

removing remaining data manually

once streaming completed, data stored in decommissioned node can removed manually described in cassandra document:

no data removed automatically node being decommissioned, if want put node service @ different token on ring, should removed manually.

this can done removing data stored in data_file_directories, commitlog_directory, , saved_caches_directory specified in cassandra.yaml file in decommissioned node.