In the previous two posts of this series (Part 1 and Part 2) I covered some of the basic commands of cassandra-stress. In this post I will start looking at the use of the stress YAML file for more advanced stress scenarios, particularly where you want to run stress against a schema that matches one you are planning to use for your application.
It’s worth noting in the intro that cassandra-stress with a YAML file use a significantly (80%?) different set of code to the standard read/write/mixed commands. So, some assumptions and learnings from the standard commands won’t hold for YAML-driven stress. To cite one example, when running based on YAML, cassandra-stress does not validate that data returned from a select has the expected values as it does with read or mixed.
For this article, I’ll reference the following YAML specification file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
# # Keyspace name and create CQL # keyspace: stressexample keyspace_definition: | CREATE KEYSPACE stressexample WITH replication = {'class': 'NetworkTopologyStrategy', 'AWS_VPC_US_WEST_2': '2'}; # # Table name and create CQL # table: eventsrawtest table_definition: | CREATE TABLE eventsrawtest ( host text, bucket_time text, service text, time timestamp, metric double, state text, PRIMARY KEY ((host, bucket_time, service), time) ) WITH CLUSTERING ORDER BY (time DESC) # # Meta information for generating data # columnspec: - name: host size: fixed(32) #In chars, no. of chars of UUID population: uniform(1..600) # We have about 600 hosts with equal events per host - name: bucket_time size: fixed(18) population: uniform(1..288) # 288 potential buckets - name: service size: uniform(10..100) population: uniform(1000..2000) # 1000 - 2000 metrics per host - name: time cluster: fixed(15) - name: state size: fixed(4) # # Specs for insert queries # insert: partitions: fixed(1) # 1 partition per batch batchtype: UNLOGGED # use unlogged batches select: fixed(10)/10 # no chance of skipping a row when generating inserts # # Read queries to run against the schema # queries: pull-for-rollup: cql: select * from eventsrawtest where host = ? and service = ? and bucket_time = ? fields: samerow # pick selection values from same row in partition get-a-value: cql: select * from eventsrawtest where host = ? and service = ? and bucket_time = ? and time = ? fields: samerow # pick selection values from same row in partition |
Before explaining the contents here, let’s see what happens when we run it with the following simple scenario:
cassandra-stress user profile=file:///eg-files/stressprofilemixed.yaml no-warmup ops(insert=1) n=100 -rate threads=1 -node x.x.x.x
After running this on an empty cluster, I ran “select count(*) from eventsrawtest;”. The result? 345 rows – probably not you would have guessed. Here’s how cassandra-stress gets to that:
- n=100 counts number of insert batches, not number of individual insert operations
- Each batch will contain 1 partition’s data (due to partitions=fixed(1) setting) and all 15 of the rows in the partition. There are 15 rows in every partition as the single cluster key (time) has a cluster setting of fixed(15). All the rows in the partition will be included in the batch due to the select: fixed(10)/10 setting (ie changing this to say fixed(5)/10 would result in half the rows from the partition being include in any given batch).
- 100 batches of 15 rows each gets you to 1500 rows so how did we end up with 345? This is due (primarily, in this case) to the relatively small range of potential values for the bucket_time. This results in a high overlap in the partition key values that end up getting generated by the uniform distributions. To demonstrate, changing the population of bucket_time to uniform(1..1288) results in 540 rows. In most cases, you want to initially insert data with no overlap to build up a base data set for testing. To facilitate this, I’ve recently submitted a cassandra-stress enhancement that provides sequential generation of seed values the same as used with the write command (https://issues.apache.org/jira/browse/CASSANDRA-12490). Changing the uniform() distribution to seq() results in the expected 1500 rows being inserted by this command.
Let’s look at some of the other column settings:
- population – determines the distribution of seed values used in the random data generation. By controlling the distribution of the seed values you control the distribution of the actual inserted values. So, for example uniform(1..100) will allow for up 100 different values each with the same chance of being selected. guassian(1..100) will also allow for up to 100 different values but as they will follow a guassian (otherwise known as normal or bell-curve) distribution, the values around the middle will have a much higher chance of being selected than the values at the extremes (so there will be a set of values more likely to get repeated and some which will occur very infrequently).
- size – determines the size (length in bytes) of the of the values created for the field.
- cluster – only applies to clustering columns, specifies the number of values for the column appearing in a single partition. The maximum number of rows in a partition is therefore the product of the maximum number of row of each clustering column (eg max(row1) * max(row 2) * max(row3)).
We covered most of the insert settings in the introductory points but here’s a recap:
- partitions: the number of different partitions to include in each generated insert batch. Once a partition is chosen for inclusion in a batch, all rows in the partition will become eligible for inclusion and then be filtered according to the select setting. Using, uniform(1..5) would result in each batches containing between 1 and 5 partitions worth of data (with an equal chance of each number in the range).
- batchtype: logged or unlogged – determines the cassandra batch type to use
- select: select determines the portion of rows from a partition to select (at random) for inclusion in particular batch. So, for example, fixed(5)/10 would include 50% of rows from the selected partition in each batch. uniform(1..10)/10 would result in between 10% and 100% of rows in the partition being included in the batch with a different select percentage being randomly picked for each partition in each batch.
The final section the yaml file that bears some explanation is the queries section. For each query, you specify:
- A name for the query (pull-for-rollup, get-a-value) which are used to refers to the queries when specifying the mix of operations through the cassandra-stress command line.
- cql – The actual query with ? characters where values from the population will be substituted in.
- fields – either samerow or multirow. For samerow, the key values to use for the select will be picked at random (following the same general population rules as for insert) for the list of row keys that has been generated for inserting. For multirow, each of the column values making up the key will be independently randomly selected so there is a chance of generating keys for the selection parameters that don’t exist in the set of data that will/could be inserted according the the population settings.
The ops command specified as part of the command line controls the mix of different operations to run. Take for example the following command:
cassandra-stress user profile=file:///eg-files/stressprofilemixed.yaml ops(insert=1, pull_for_rollup=1, get-value=10) n=120 -node x.x.x.x
This will execute insert batches, pull_for_rollup queries and get-value queries in the ratio 1:1:10. So for this specific example, we’d get 10 inserts, 10 pull_for_rollup queries and 100 get-value queries.
Hopefully that’s explained the key information you need to use a YAML profile for running cassandra stress. In future instalments I’ll take a look at some of the remaining command line options and walk through a full end-to-end example of designing and executing a test.
Catch my full presentation at the Cassandra Summit 2016, in San Jose. See the schedule for my ‘Load Testing Cassandra Applications’ presentation here.
Click here for Part One: Deep Diving into Cassandra Stress
Click here for Part Two: Mixed Command
When I want to execute the command there comes a NoHostAvailableException. But the Nodes are all up and normal, as shown with nodetool status.
Hi Jessica,
You can see that happen if you overload the cluster. I would try starting with a very low load (thread count) and see if it runs reliably. If you see this under low load it could indicate a problem with configuration of your cluster.
Cheers
Ben
Hi Ben,
Would seq() generate sequential values, or sequential seeds that will result in random values (but with the expected number of unique numbers ).
I’m interested in using this to generate time-series data where rows are interested with an increasing timestamp. I’m using TWCS compaction, so the order of insertion actually matters.
Cheers,
Eugene
Hi Eugene,
It generates sequential seeds so you will still get random values as you suspected.
I can see that actual sequential values would be useful for testing TWCS. One thought – it might be possible to get what you want by using the now() function with a timeuuid in your insert (not sure how this will play with cassandra-stress but might be worth a try).
Cheers
Ben
Hi Ben,
Got it more or less working by running a query that looks like
cql: insert into test_table ( customer_id, time, test_data) VALUES(? , toTimestamp(now()), ? )
It works well if I run it with a relatively large number of executions and a fairly small distribution for the primary key (for example: n=1000000, population: norm(0..1000)) – I end up getting a nice normal distribution for the number of rows per partition.
However, want to initialize the table with 1 row per partition, and trying to use seq for that. My yaml has,
columnspec:
– name: customer_id
size: fixed(64)
population: seq(1..1000)
And I run it with the following options:
n=1000 ops\(insertq=1\) no-warmup truncate=once -rate threads=1
When I check C* with cqlsh I all my keys are negative somewhere in the range of 0-999 (but are not sequential, for example – there are 2 rows for -105, but zero for -106)
Any ideas what’s going on?
I think this comment from Benedict on the seq() JIRA ticket is probably what’s impacting you:
===============
partition keys are a distinct beast, and if your population distribution for these is tiny then yes you will get overwrites, and I’m really not sure there’s anything we can reliably do about that. Mostly I’ve been talking about behaviour within a partition (except when pointing out some breakages).
The command line “-pop” property specifies the population of unique partition seeds. These have to be translated into the partition key population distribution(s) first, which then between them uniquely produce the partition’s contents (different seeds hitting the same PK will produce the same entire partition). The problem is that the size of the unique seed set could be gigantic (we let n be billions in size, and it is often necessary to run with datasets this large), so enumerating all of these unique seeds and determining their value in the partition key column population distributions would be prohibitively expensive. So we just accept that users should sensibly ensure their partition key population distribution is large enough to accommodate enough random samples to fulfil their seed population.
============
TL;DR; partition keys are a treated a bit differently (and a bit unexpectedly to me)
I don’t know that I’ve played around with it that much but maybe try using SEQ() with the -pop argument.
Cheers
Ben
Tried it it with the -pop argument and still getting all negative numbers. How are you using it? I thought that the main use case was to generate a known sequence of partition keys to get the expected number of rows (unless you are using it on the clustering key, but then the number of partitions will be random)
Yes, I’ve mainly used it with clustering keys and with the purpose of filling up disks for background data load quickly rather than trying to generate a specific number of rows. TBH, it’s a while since I’ve looked at this. I’ve got another cassandra-stress task I want to get back to so I’ll try to find some time to take a look at this in the near future and see if I can find any further info.
Cheers
Ben
Hi,
I did some testing of this. Ran with the following arguments:
user profile=stresssimple5.yaml no-warmup ops(insert=1) n=10000 -rate threads=10 -mode native cql3 protocolVersion=3 -node 127.0.0.1
And the following YAML:
#
# Table name and create CQL
#
table: test5
table_definition: |
CREATE TABLE test5 (
pk int,
val text,
PRIMARY KEY (pk)
)
columnspec:
– name: pk
size: fixed(64)
population: seq(1..100000)
#
# Specs for insert queries
#
insert:
partitions: fixed(1) # 1 partition per batch
batchtype: UNLOGGED # use unlogged batches
select: fixed(10)/10 # no chance of skipping a row when generating insert
In all my tests I got the expected number of rows inserted (ie whatever was specified for n=). The actual values for the PK are random (and all negative for some reason) which is what I would expect. The actual values don’t really make much difference for the partition key given it’s hashed (the number of different values does make a difference and is what seq() is aimed to help you control).
I can see you might want to generate sequential values for clustering key fields seeing as ordering is important there but as far as I know there is not currently anyway of doing that with cassandra-stress.
Cheers
Ben
Thanks for more excellent posts. When doing a custom run with your yaml, I get 8 “results” sections (as opposed to just 1 for write/mixed, even if I have many threads). Any idea why, and how to interpret the results?
I’m guessing you’re running without specifying the number of threads (or similar rate limiter). When you do this cassandra-stress automatically does multiple runs with different number of threads until the throughput stops significantly increasing (I think – definitely multiple runs but I’m my memory is a bit hazy on when it stops).