3 answers
- 10-1
I have made another work around for cqlsh failures. Consider this as a failsafe for different environments. Again we have to increase the ReadinessProbe value for aaf-cs pods to 600 seconds (varies by setup)
In push.sh
for T in $(ls *.dat); do
if [ -s $T ]; then
until cqlsh --request-timeout=3600 -e "COPY authz.${T/.dat/} FROM '$T' WITH DELIMITER='|';";
do
echo "Hit failure...sleep for 10 seconds and try again"
sleep 10
done
fi
done
Also increased the write_request_timeout_in_ms: 20000 in order to prevent coordinator failures
In cmd.sh function install_cql
function install_cql {
wait_start started
# Now, make sure data exists
if [ ! -e $INSTALLED_VERSION ] && [ -n "$(/usr/bin/cqlsh -e 'describe keyspaces' | grep authz)" ]; then
/usr/bin/cqlsh -e 'DROP KEYSPACE authz'
fi
if [ -z "`/usr/bin/cqlsh -e 'describe keyspaces' | grep authz`" ]; then
status install
echo "Initializing Cassandra DB"
echo "Docker Installed Basic Cassandra on aaf_cass. Executing the following "
echo "NOTE: This creator provided is only a Single Instance. For more complex Cassandra, create independently"
echo ""
echo " cd /opt/app/aaf/cass_init"
until /usr/bin/cqlsh -e 'describe keyspaces';
do
echo "Casandra DB is not up yet!! Sleeping for 60 seconds"
sleep 30
done
cd /opt/app/aaf/cass_init
echo " cqlsh -f keyspace.cql"
/usr/bin/cqlsh --request-timeout=3600 -f keyspace.cql
status keyspace installed
echo " cqlsh -f init.cql"
/usr/bin/cqlsh --request-timeout=3600 -f init.cql
status data initialized
echo ""
echo "The following will give you a temporary identity with which to start working, or emergency"
echo " cqlsh -f temp_identity.cql"
echo "casablanca" > $INSTALLED_VERSION
else
echo "Cassandra DB already includes 'authz' keyspace"
fi
status $1
}
Now I am able to deploy AAI without failures.
Add your comment... - 10-1
I have found the root cause for this problem.
This is a timing issue
onap)
cd /opt/app/aaf/cass_init
# start install_onap (which calls install_cql first) in background, waiting for process to start
install_onap &
# Startup like normal
echo "Cassandra Startup"
/usr/local/bin/docker-entrypoint.sh
By the time we run
echo " cd /opt/app/aaf/cass_init"
cd /opt/app/aaf/cass_init
echo " cqlsh -f keyspace.cql"
/usr/bin/cqlsh -f keyspace.cql
status keyspace installed
echo " cqlsh -f init.cql"
/usr/bin/cqlsh -f init.cql
Casandra was not listening for cql clients (port 9042 was not opened).
Perhaps we should add a check here to see if 9042 was opened
function wait_start {
sleep 10
status wait for cassandra to start
for CNT in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do
if [ -z "$(grep 'listening for CQL clients' /var/log/cassandra/system.log)" ]; then
echo "Waiting for Cassandra to start... Sleep 10"
sleep 10
else
break
fi
done
}
So, I have added a minute delay before running /usr/bin/cqlsh -f keyspace.cql as a work around and it worked
echo " cqlsh -f keyspace.cql"
until /usr/bin/cqlsh -e 'describe keyspaces';
do
echo "Casandra is not up yet!! Sleeping for 60 seconds"
sleep 60
done
/usr/bin/cqlsh -f keyspace.cql
however I am limited with 1Gbps link here and will upgrade to 10Gb and give a try again.Add your comment... - 10-1
If the “authz” keyspace is not found, then the Cassandra instance either was not created, or the data volume was removed.
If you can log onto the “aaf-cass” container,
- cd to /opt/app/aaf/cass_init
- run cqlsh
- type
describe keyspaces
if “authz” isn’t one of them,
- type
source ‘init.cql’
that will create the authz tables
- exit cqlsh
- the data will need to be loaded:
- bash prep.sh
- bash push.sh
At this point, you should be able to use CQLSH to see some data
cqlsh -k authz
cqlsh> select * from ns;
cqlsh> select * from cred;
cqlsh> exit
if you can do this, then the Data is now ready for AAF to use
Add your comment...
Aaf-service logs:
2019-02-19 14:07:21,855 WARN [init] 2019-02-19T14:07:21.855+0000 INIT [init] Cassandra is using Default Policy, which is not DC aware
2019-02-19 14:07:27,463 WARN [init] 2019-02-19T14:07:27.463+0000 INIT [init] Instantiating DAOs
com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'authz' does not exist
at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:49)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.Cluster.connect(Cluster.java:334)
at org.onap.aaf.auth.dao.AbsCassDAO.getSession(AbsCassDAO.java:434)
at org.onap.aaf.auth.dao.AbsCassDAO.getSession(AbsCassDAO.java:394)
at org.onap.aaf.auth.dao.cass.CacheInfoDAO.init(CacheInfoDAO.java:333)
at org.onap.aaf.auth.dao.cass.CacheInfoDAO.<init>(CacheInfoDAO.java:83)
at org.onap.aaf.auth.dao.hl.Question.<init>(Question.java:153)
at org.onap.aaf.auth.service.AAF_Service.<init>(AAF_Service.java:99)
at org.onap.aaf.auth.service.AAF_Service.main(AAF_Service.java:229)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Keyspace 'authz' does not exist
at com.datastax.driver.core.Responses$Error.asException(Responses.java:181)
at com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:215)
aaf-cs logs:
INFO [OptionalTasks:1] 2019-02-19 00:11:57,848 CassandraRoleManager.java:356 - Created default superuser role 'cassandra'
INFO [main] 2019-02-19 00:11:58,452 NativeTransportService.java:70 - Netty using native Epoll event loop
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
keyspace installed
cqlsh -f init.cql
INFO [main] 2019-02-19 00:12:01,752 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO [main] 2019-02-19 00:12:01,849 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO [main] 2019-02-19 00:12:04,051 CassandraDaemon.java:529 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
init.cql:6:InvalidRequest: Error from server: code=2200 [Invalid query] message="Keyspace 'authz' does not exist"
init.cql:23:InvalidRequest: Error from server: code=2200 [Invalid query] message="No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename"
init.cql:24:InvalidRequest: Error from server: code=2200 [Invalid query] message="No keyspace has been specified. USE a keyspace, or explicitly specify keyspace.tablename"
The following will give you a temporary identity with which to start working, or emergency
cqlsh -f temp_identity.cql
initialized
prep data for bootstrapping
prep data
push data to cassandra
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
<stdin>:1:Keyspace 'authz' not found.
/opt/app/aaf/cass_init
ready
WARN [Service Thread] 2019-02-19 00:52:23,648 GCInspector.java:282 - ParNew GC in 288804ms. CMS Old Gen: 9256176 -> 20744144; Par Eden Space: 429522944 -> 0; Par Survivor Space: 42182152 -> 46077880
I am seeing the same behavior using master branch as well. Any help would be appreciated.