I have a cassandra table with around 50 columns. How do I get data from cassandra source table and save it in cassandra target table using spark RDD?
I tried a small example which is allowing me to mention only 12 columns. When I add a 13th column it is raising an error.
Can somebody help on this?
sc.cassandraTable("ks_test","test_status")
.select("TEST1","TEST2","TEST3","TEST4","TEST5","TEST6","TEST7","TEST8","TEST9",
"TEST10","TEST11","TEST12")
.as((TEST1:String,TEST2:String,TEST3:String,TEST4:String,TEST5:String,TEST6:String,TEST7:String,
TEST8:String,TEST9:String,TEST10:String,TEST11:String,TEST12:String) =>
("Jun-2016","2016-06-11",TEST1,TEST2,TEST3,TEST4,TEST5,TEST6,TEST7,TEST8,TEST9,
TEST10,TEST11,TEST12))
.saveToCassandra("ks_test","test_status_copy",
SomeColumns("test_bucket","test_creation_date","TEST1","TEST2","TEST3","TEST4","TEST5","TEST6","TEST7",
"TEST8","TEST9","TEST10","TEST11","TEST12"
))
13th Column Code :
sc.cassandraTable("ks_test","test_status") .select("TEST1","TEST2","TEST3","TEST4","TEST5","TEST6","TEST7","TEST8","TEST9", "TEST10","TEST11","TEST12","TEST13") .as((TEST1:String,TEST2:String,TEST3:String,TEST4:String,TEST5:String,TEST6:String,TEST7:String, TEST8:String,TEST9:String,TEST10:String,TEST11:String,TEST12:String,TEST13:String) => ("Jun-2016","2016-06-11",TEST1,TEST2,TEST3,TEST4,TEST5,TEST6,TEST7,TEST8,TEST9, TEST10,TEST11,TEST12,TEST13)) .saveToCassandra("ks_test","test_status_copy", SomeColumns("test_bucket","test_creation_date","TEST1","TEST2","TEST3","TEST4","TEST5","TEST6","TEST7", "TEST8","TEST9","TEST10","TEST11","TEST12","TEST13" ))
Error :
:56: error: overloaded method value as with alternatives: [B, A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11](f: (A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11) => B)(implicit evidence$81: scala.reflect.ClassTag[B], implicit evidence$82: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$83: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$84: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$85: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$86: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$87: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$88: com.datastax.spark.connector.types.TypeConverter[A6], implicit evidence$89: com.datastax.spark.connector.types.TypeConverter[A7], implicit evidence$90: com.datastax.spark.connector.types.TypeConverter[A8], implicit evidence$91: com.datastax.spark.connector.types.TypeConverter[A9], implicit evidence$92: com.datastax.spark.connector.types.TypeConverter[A10], implicit evidence$93: com.datastax.spark.connector.types.TypeConverter[A11])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10](f: (A0, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10) => B)(implicit evidence$69: scala.reflect.ClassTag[B], implicit evidence$70: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$71: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$72: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$73: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$74: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$75: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$76: com.datastax.spark.connector.types.TypeConverter[A6], implicit evidence$77: com.datastax.spark.connector.types.TypeConverter[A7], implicit evidence$78: com.datastax.spark.connector.types.TypeConverter[A8], implicit evidence$79: com.datastax.spark.connector.types.TypeConverter[A9], implicit evidence$80: com.datastax.spark.connector.types.TypeConverter[A10])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5, A6, A7, A8, A9](f: (A0, A1, A2, A3, A4, A5, A6, A7, A8, A9) => B)(implicit evidence$58: scala.reflect.ClassTag[B], implicit evidence$59: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$60: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$61: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$62: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$63: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$64: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$65: com.datastax.spark.connector.types.TypeConverter[A6], implicit evidence$66: com.datastax.spark.connector.types.TypeConverter[A7], implicit evidence$67: com.datastax.spark.connector.types.TypeConverter[A8], implicit evidence$68: com.datastax.spark.connector.types.TypeConverter[A9])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5, A6, A7, A8](f: (A0, A1, A2, A3, A4, A5, A6, A7, A8) => B)(implicit evidence$48: scala.reflect.ClassTag[B], implicit evidence$49: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$50: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$51: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$52: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$53: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$54: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$55: com.datastax.spark.connector.types.TypeConverter[A6], implicit evidence$56: com.datastax.spark.connector.types.TypeConverter[A7], implicit evidence$57: com.datastax.spark.connector.types.TypeConverter[A8])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5, A6, A7](f: (A0, A1, A2, A3, A4, A5, A6, A7) => B)(implicit evidence$39: scala.reflect.ClassTag[B], implicit evidence$40: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$41: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$42: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$43: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$44: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$45: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$46: com.datastax.spark.connector.types.TypeConverter[A6], implicit evidence$47: com.datastax.spark.connector.types.TypeConverter[A7])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5, A6](f: (A0, A1, A2, A3, A4, A5, A6) => B)(implicit evidence$31: scala.reflect.ClassTag[B], implicit evidence$32: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$33: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$34: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$35: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$36: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$37: com.datastax.spark.connector.types.TypeConverter[A5], implicit evidence$38: com.datastax.spark.connector.types.TypeConverter[A6])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4, A5](f: (A0, A1, A2, A3, A4, A5) => B)(implicit evidence$24: scala.reflect.ClassTag[B], implicit evidence$25: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$26: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$27: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$28: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$29: com.datastax.spark.connector.types.TypeConverter[A4], implicit evidence$30: com.datastax.spark.connector.types.TypeConverter[A5])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3, A4](f: (A0, A1, A2, A3, A4) => B)(implicit evidence$18: scala.reflect.ClassTag[B], implicit evidence$19: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$20: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$21: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$22: com.datastax.spark.connector.types.TypeConverter[A3], implicit evidence$23: com.datastax.spark.connector.types.TypeConverter[A4])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2, A3](f: (A0, A1, A2, A3) => B)(implicit evidence$13: scala.reflect.ClassTag[B], implicit evidence$14: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$15: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$16: com.datastax.spark.connector.types.TypeConverter[A2], implicit evidence$17: com.datastax.spark.connector.types.TypeConverter[A3])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1, A2](f: (A0, A1, A2) => B)(implicit evidence$9: scala.reflect.ClassTag[B], implicit evidence$10: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$11: com.datastax.spark.connector.types.TypeConverter[A1], implicit evidence$12: com.datastax.spark.connector.types.TypeConverter[A2])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0, A1](f: (A0, A1) => B)(implicit evidence$6: scala.reflect.ClassTag[B], implicit evidence$7: com.datastax.spark.connector.types.TypeConverter[A0], implicit evidence$8: com.datastax.spark.connector.types.TypeConverter[A1])com.datastax.spark.connector.rdd.CassandraRDD[B] [B, A0](f: A0 => B)(implicit evidence$4: scala.reflect.ClassTag[B], implicit evidence$5: com.datastax.spark.connector.types.TypeConverter[A0])com.datastax.spark.connector.rdd.CassandraRDD[B] cannot be applied to ((String, String, String, String, String, String, String, String, String, String, String, String, String) => (String, String, String, String, String, String, String, String, String, String, String, String, String, String, String)) .as((ac_reg_no:String,work_pkg_id:String,a_c_egnr_suprvsr:String,a_c_mechanics:String,ac_rank:String,ac_type:String,acting_base_shift_mgr:String, ^
Aucun commentaire:
Enregistrer un commentaire