database administration - Cassandra: Removing a node -
i remove node cassandra cluster , following these 2 related questions (here , here) , cassandra document. still not quite sure exact process.
my first question is: following way remove node cassandra cluster correct?
decommission
node remove.removetoken
node decommissioned.
if above process right, how can tell decommission process completed can proceed second step? or safe step 2 right after step 1?
in addition, cassandra document says:
you can take node out of cluster nodetool decommission live node, or nodetool removetoken (to other machine) remove dead one. assign ranges old node responsible other nodes, , replicate appropriate data there. if decommission used, data stream decommissioned node. if removetoken used, data stream remaining replicas.
no data removed automatically node being decommissioned, if want put node service @ different token on ring, should removed manually.
does mean decommissioned node dead node? in addition, no data removed automatically node being decommissioned, how can tell when safe remove data decommissioned node (i.e., how know when data-streaming completed?)
removing node cassandra cluster should following steps (in cassandra v1.2.8):
- decommission target node
nodetool -host node decmmission
. - once data streaming decommissioned node completed, manually delete data in decommissioned node (optional).
update: above process works seed nodes. in such case, cluster still able run smoothly without requiring restart. when need restart cluster other reasons, sure update seeds
parameter specified in cassandra.yaml
nodes.
decommission target node
when decommission starts, decommissioned node first labeled leaving
(marked l
). in following example, remove node-76
:
> nodetool -host node-76 decommission > nodetool status status=up/down |/ state=normal/leaving/joining/moving -- address load tokens owns host id rack un node-70 9.79 gb 256 8.3% e0a7fb7a-06f8-4f8b-882d-c60bff51328a 155 un node-80 8.9 gb 256 9.2% 43dfc22e-b838-4b0b-9b20-66a048f73d5f 155 un node-72 9.47 gb 256 9.2% 75ebf2a9-e83c-4206-9814-3685e5fa0ab5 155 un node-71 9.48 gb 256 9.5% cdbfafef-4bfb-4b11-9fb8-27757b0caa47 155 un node-91 8.05 gb 256 8.4% 6711f8a7-d398-4f93-bd73-47c8325746c3 155 un node-78 9.11 gb 256 9.4% c82ace5f-9b90-4f5c-9d86-0fbfb7ac2911 155 ul node-76 8.36 gb 256 9.5% 15d74e9e-2791-4056-a341-c02f6614b8ae 155 un node-73 9.36 gb 256 8.9% c1dfab95-d476-4274-acac-cf6630375566 155 un node-75 8.93 gb 256 8.2% 8789d89d-2db8-4ddf-bc2d-60ba5edfd0ad 155 un node-74 8.91 gb 256 9.6% 581fd5bc-20d2-4528-b15d-7475eb2bf5af 155 un node-79 9.71 gb 256 9.9% 8e192e01-e8eb-4425-9c18-60279b9046ff 155
when decommissioned node marked leaving
, streaming data other living nodes. once streaming completed, node not observed ring structure, , data owned other nodes increase:
status=up/down |/ state=normal/leaving/joining/moving -- address load tokens owns host id rack un node-70 9.79 gb 256 9.3% e0a7fb7a-06f8-4f8b-882d-c60bff51328a 155 un node-80 8.92 gb 256 9.6% 43dfc22e-b838-4b0b-9b20-66a048f73d5f 155 un node-72 9.47 gb 256 10.2% 75ebf2a9-e83c-4206-9814-3685e5fa0ab5 155 un node-71 9.69 gb 256 10.6% cdbfafef-4bfb-4b11-9fb8-27757b0caa47 155 un node-91 8.05 gb 256 9.1% 6711f8a7-d398-4f93-bd73-47c8325746c3 155 un node-78 9.11 gb 256 10.5% c82ace5f-9b90-4f5c-9d86-0fbfb7ac2911 155 un node-73 9.36 gb 256 9.7% c1dfab95-d476-4274-acac-cf6630375566 155 un node-75 9.01 gb 256 9.5% 8789d89d-2db8-4ddf-bc2d-60ba5edfd0ad 155 un node-74 8.91 gb 256 10.5% 581fd5bc-20d2-4528-b15d-7475eb2bf5af 155 un node-79 9.71 gb 256 11.0% 8e192e01-e8eb-4425-9c18-60279b9046ff 155
removing remaining data manually
once streaming completed, data stored in decommissioned node can removed manually described in cassandra document:
no data removed automatically node being decommissioned, if want put node service @ different token on ring, should removed manually.
this can done removing data stored in data_file_directories
, commitlog_directory
, , saved_caches_directory
specified in cassandra.yaml
file in decommissioned node.
Comments
Post a Comment